AI News: CogvideoX-ControlNet and Veo by Google DeepMind and More

Hey everyone! Here’s what’s new in AI today:

CogvideoX-ControlNet

There’s a new tool for people who make videos called “CogvideoX-ControlNet.” It helps you use the CogvideoX model, which is very strong and can turn pictures into short videos. So, you can change a single picture into a fun video! It’s open-source, so you can check it out and maybe even help make it better on GitHub. https://github.com/TheDenk/cogvideox-controlnet

Meta Movie Gen Now Has Audio

This one is super cool for video makers. “Meta Movie Gen” can now create sound that matches your videos! You just give it a video and maybe a few text prompts, and it can add background sounds, music, or even ambient noise, making your video feel more real.

https://ai.meta.com/blog/movie-gen-media-foundation-models-generative-ai-video/?utm_source=twitter&utm_medium=organic_social&utm_content=video&utm_campaign=moviegen

As part of Meta Movie Gen, we trained a 13B parameter audio generation model that can take a video + optional text prompts to generate high quality audio — including ambient sound, foley & instrumental background music — all synced to the video. Details ➡️ https://t.co/trp3ekFgMP pic.twitter.com/aw4DasBxEI

— AI at Meta (@AIatMeta) October 8, 2024

Veo by Google DeepMind

Google DeepMind just released a video of their new AI called “Veo.” It’s their most advanced video creation tool so far. You can see it in action on their website.

https://deepmind.google/technologies/veo

FLUX.1-Dev ControlNet Inpainting

“FLUX.1-dev ControlNet Inpainting” helps fix or fill in parts of an image. It’s especially useful if you’ve got an image with missing or damaged spots because it can recreate those parts seamlessly.

https://huggingface.co/alimama-creative/FLUX.1-dev-Controlnet-Inpainting-Beta

Diffusers Image Fill With Prompt

This framework helps remove unwanted sections from images. You can test it out on Hugging Face’s demo page and see how it works.

https://huggingface.co/spaces/ameerazam08/diffusers-image-fill-with-prompt

Open NotebookLM

This is neat if you like podcasts! “Open NotebookLM” can turn any PDF into a podcast by reading it out as a conversation between two voices. It’s a cool way to listen to documents instead of reading them.

https://github.com/gabrielchua/open-notebooklm/tree/main

LM Studio 0.3.4

If you’re using a Mac, “LM Studio 0.3.4” can help you run large language models (LLMs) super fast. It uses Apple’s MLX to make the models run better on Apple’s Silicon Macs.

https://github.com/lmstudio-ai/mlx-engine

LM Studio 0.3.4 ships with Apple MLX 🚢🍎

Run on-device LLMs super fast, 100% locally and offline on your Apple Silicon Mac!

Includes:

> run Llama 3.2 1B at ~250 tok/sec (!) on M3
> enforce structured JSON responses
> use via chat UI, or from your own code
> run multiple… pic.twitter.com/XQ32Bp0Fcv

— LM Studio (@LMStudioAI) October 8, 2024

PocketPal

Now you can run AI on your phone! “PocketPal” lets you use Llama 3.2 completely offline, right on your smartphone. It’s handy for quick AI tasks when you’re not online.

https://apps.apple.com/us/app/pocketpal-ai/id6502579498

That’s it for today’s AI updates!