CE.SDK – IMG.LY Blog

Evaluating CoDesign MCP performance in resizing

Dustin — Fri, 21 Aug 2026 09:00:00 GMT

You designed a banner. The same day, you need it in five sizes, and none of them can be wrong.

We handed that job to six AI models through CoDesign MCP: take this approved design and rebuild it for five ad formats. All six came back with ads you could ship. Gemini 3.7 Flash did it for thirty cents and passed every check we wrote.

That is what CoDesign MCP is for. You point an AI assistant such as Claude at your approved design, name the formats you need, and it opens the file, rebuilds the layout for each one and exports production files. What comes back is an editable design file, not a picture of one.

Speed and price are the easy things to measure. We wanted to measure quality too, so we built an eval that runs the job automatically, opens every result and reports what passed. This is the first scenario out of that suite.

Three banners, one brief

Three different models were handed the same 320×50 slot, cut from the same master creative against the same brief, and all three came back with an ad you could traffic tomorrow. Every one of them keeps the wordmark and the button, nothing is clipped, and no type is set below 11px.

The 300×250 they came from carried five things: the wordmark, the headline, a retention chart, the offer line and the button. At 320×50 the wordmark and the button survive and the chart cannot, which leaves room for exactly one line of copy, so each model had to choose between the headline and the offer.

One dropped the headline and kept the offer, “14-day free trial”.
One kept the headline but cut “your users” down to “users” to buy itself the line.
One kept the headline whole across two lines and dropped the offer.

We have opinions about which of those reads best, the way any design team would. What we do not have is a way to prove one of them correct, because the choice depends on whether this campaign is selling the product or the trial, and that lives in the media brief rather than in the design. Our checks can confirm that all three are valid and that none of them are broken; they cannot tell you which one the campaign actually needed. Grading the half that has a right answer and leaving the other half to the people who own the brief is the distinction our evals are built around.

What is gradeable and what is not

The obvious thing to evaluate is the thing that demos well, prompt in and poster out, but we were more interested in the work that actually fills a designer’s week, which is mostly derivative. The design decisions were made once, at the master, and everything after that is a matter of carrying them faithfully into thirty other shapes. That is more technical work than creative work, which is exactly why it is worth handing to an agent.

Ask which of the finished designs is better and you have a question with no ground truth to appeal to, since human raters disagree with each other and LLM judges reliably prefer their own output. You could ask a lot of humans and rank the results, but that is not something you can run on every build. Wrong designs, though, are wrong in ways you can query: text sitting behind another block, content off the page, type under the legibility floor. Part of design quality is judgement and part of it is fact, and what we have built grades the second part, which means everything below is an automated result rather than a design review. A model can pass every check we wrote and still make an ugly ad.

What we are doing right now

When a model finishes a scenario, programmatic checks open the result and ask whether it is plausibly good. That starts with whether the run produced every design artifact and .imgly file at the right sizes, and the rest depends on the scenario: in a localization run we also check that the words were actually replaced.

This is a technical floor rather than a grade. LLM judges are still unreliable enough that we do not use one on output quality, because if it is hard for us as designers to say which design is more correct, it is harder for a model, and the result would be noise.

In the future we plan to experiment more with such methodologies but we also do not want to rely on the false security of an extremely noisy grading process.

The comparison view for the display-banner matrix: cost and time sitting directly above what each model actually produced. The 1/1 counts runs, one repeat per cell, not check scores.

One master, five ad sizes

The scenario is a display campaign, because that is the version of this problem customers actually have: the same size matrix, every campaign. The master is a 300×250 MPU for ATLAS, an imaginary product-analytics SaaS running a free-trial campaign, carrying a wordmark, the headline “See what your users actually do”, a retention bar chart labelled “43% retained”, the line “14-day free trial. No card.”, and a “Start free trial” button.

The 300×250 master at the top, and the four other sizes on the media plan cut from it. ATLAS is our own fixture, not a real customer.

The brief asks for the full matrix: 300×250, 728×90, 160×600, 300×600 and 320×50. Those sizes span a 16:1 swing in aspect ratio, so the right answer at each one is a genuinely different composition, and at some of them it means taking content out rather than shrinking it.

Ten checks grade the result and two of them do the real work. 728×90: the chart was dropped, not shrunk kills the model that squashes a bar chart into a 90-pixel strip, which satisfies every geometric constraint in the brief while being completely wrong. the five sizes are different layouts kills the model that scales one composition five ways and calls it a matrix.

The master first, then the six 728×90 leaderboards. Every model that delivered deleted the retention chart.

Every model that delivered arrived at the same composition, wordmark left, headline across the middle, offer underneath it, button on the right and the retention chart deleted, which is six models from five different vendors reaching the same answer independently. We did not expect that, and it is still the result we find hardest to stop thinking about.

The same six at 160×600, master first. With vertical room they all kept the chart. The flat format is where the decision to drop it had to be made.

Which model is best for using with CoDesign MCP?

#	model	checks	model spend	time	tool calls
1	gemini-3.7-flash	10 / 10	$0.30	6.7 min	37
2	glm-5.2	10 / 10	$0.40	8.2 min	25
3	gpt-5.6-sol	10 / 10	$0.52	4.0 min	35
4	grok-4.6	10 / 10	$1.20	20.9 min	39
5	claude-fable-5	10 / 10	$3.95	10.3 min	26
6	claude-opus-5	10 / 10	$3.97	18.8 min	40

Running this scenario illustrates general numbers across our eval suite. The 6 models above are consistently able to follow the prompts to produce different output designs as requested in the scenario. These numbers also highlight quite the spread in model costs and time spend. What you do not see in these numbers is that glm 5.2, gpt-5.6-sol and grok 4.6 are producing suboptimal designs in our opinion. They are valid, but subjectively fable, opus and gemini 3.7 consistently across runs and across scenarios perform simply better.

The biggest surprise for us is Gemini 3.7 Flash, which produced the whole five-size matrix for $0.30 in model spend, in under seven minutes. At thirty cents a designer opens six composed variants instead of a blank artboard and takes the best one further, which changes what the tool is for.

Meanwhile there were no surprises regarding Opus 5.0 and Fable 5.0, both models perform really well - but are slow and expensive.

What does it mean for CoDesign MCP

You can now resize a master design into 5 different sizes reliably for $0.30 using Gemini 3.7 Flash and the CoDesign MCP.

If you are on a Claude Subscription plan, you might just want to continue using your Claude Code with the CoDesign MCP since it offers the best overall design quality - even if it’s slow and somewhat expensive.

We will continue to work on upgrading and improving measurements of how well CoDesign MCP works with different models, looking into other scenarios like localization (translating and changing a design for a different locality and language) as well as rebranding existing design assets.

You can install CoDesign MCP at img.ly/codesign.

CE.SDK v1.78 Release Notes

Neslihan — Tue, 21 Jul 2026 12:00:00 GMT

CE.SDK v1.78 is a typography and reliability release. It adds a dedicated path control for curved text on the web, alongside a broad set of engine and editor fixes across web, mobile, and backend.

Let’s dive in.

More Expressive Typography

Curve Text with a Dedicated Path Control on the Web

The web editor gains a Path control for text blocks. Users can place text on a curve using Circle, Arch, and Wave presets, then adjust position, direction, and offset directly in the inspector.

For custom shapes, Edit Path lets users draw their own baseline instead of picking a preset, so text can follow whatever curve a design calls for. This builds on the text-on-path support added in v1.77.

→ Edit Text Documentation (Web)

Improvements

Keep Edited Text Above the Mobile Keyboard

On mobile web, the canvas reserves space for the on-screen keyboard and pans to the text cursor, so the text being edited stays visible while typing.

Browse Typefaces Smoothly on Slow Networks

The web typeface library shows font names immediately and loads previews lazily as rows scroll into view, keeping the picker responsive on low-end devices.

Refresh iOS Asset Libraries Automatically

On iOS, an open library updates in place when its source’s contents change, matching the web editor.

Stability and Fixes

This release includes a broad set of stability, performance, and rendering fixes across the engine and editors. Among them: a Safari video-export fix for projects that contain uploaded video, faster handling of large caption files, curved-text caret and selection fixes, iOS effect-sheet and web keyboard-accessibility fixes, and archive-loading crash guards. See the changelog for the full list.

Changelog

All improvements, breaking changes, and fixes are in the v1.78.0 changelog.

Thank you for building with IMG.LY.

13 Questions Teams Ask Themselves Before Choosing IMG.LY

Marc — Fri, 19 Jun 2026 09:00:00 GMT

Most teams evaluating an in-app editor arrive with the same shortlist of questions. These are the 13 that come up most across 600+ customers and 28 structured interviews, each answered straight, with the numbers behind it.

Why integrate an editor instead of building our own?

A production-ready SDK turns months of editor engineering into a couple of weeks of integration. Building an equivalent editor in-house runs 6 to 12 months and ties up two or three senior developers. The median IMG.LY customer reaches production within just 14 days, roughly 13× faster than the in-house route. For fast-moving companies especially, staying on your core is what pays off. Time spent on the editor is time not spent on what actually sets your product apart.

Can’t we just use an open-source library like Fabric.js or Konva?

You can, and some companies do. The problems start once you need more than a drawing surface. Open-source canvas libraries give you that surface and little else, no text on a path, no video timeline, no background removal, no cross-platform parity, and no ready-made UI. You build all of that yourself, and then you own the maintenance forever. The library is free, the year of senior engineering it takes to reach production quality is not.

It was very buggy, lack of features, not looking good, and we decided to not go deeper.

Engineering lead at a company that evaluated open source before CE.SDK

CE.SDK gives you the engine and the UI in one SDK, with the same rendering across web, iOS, and Android. If you want the full breakdown, we put open source next to building and buying in our Fabric.js comparison.

How fast can we realistically be in production?

A working prototype usually takes about two days, and a production-ready integration around two weeks. That time goes into wiring the existing SDK and API into your product and styling it to fit. Our AI developer tooling accelerates the process even more, giving your AI coding assistant full CE.SDK context for the integration. For fast-moving companies, this can be a real booster. Plai went from deal to live in about a month, then quickly scaled to thousands of ad campaigns with the new editor in place.

Every design edit request lands on my team. How do we scale that?

You stop being the bottleneck. Instead of your team producing every asset, users edit themselves inside your product, so output scales with your user base, not your headcount.

When you're dealing with a graphic designer who has a queue of work, you usually need to wait.

Product lead at an IMG.LY customer

The digital agency Digitas built its dealer portal on CE.SDK, and removing that bottleneck led to 85% of videos being created fully in-app, with active users making 3× more than before. None of that work routes through a design team.

The same engine also scales without anyone editing at all. Run it headless and it generates assets automatically. One template plus your data renders thousands of on-brand variants through the API. PostBuddy, for example, does exactly this. Its users choose a template, upload a CSV, and the engine auto-generates thousands of personalized, print-ready postcards, with no one editing a single one. Self-serve covers the one-offs, automation covers the scale.

Our users aren’t designers. How do we keep results on-brand without manual review?

With guardrails you set. Templates, locked brand elements, and constraints let a non-designer produce polished, on-brand work by default, because the editor only exposes what you allow. The output can even beat the professionals.

I built a better-looking asset in the portal than the agency ever did.

Martin Röhr — Senior Project Manager, Digitas

Nobody has to check the output later, because the guardrails already did.

We already have an editor. Is replacing it worth the costs?

The editing experience is part of the sale. When the flow is smooth, more users finish what they start, and completion is conversion. A clunky editor quietly costs you sales.

"Our brand didn't match our experience. You could see it in our bounce rates," says Tom Rowe, and the difference is easy to spot: "you can tell when competitors are using the old platform just by looking at it."

Tom Rowe — Product & Engineering, The Print Bar

When the status quo quietly costs deals, the editor should not be treated as a nice-to-have, but as an essential.

Will it match our product’s look and feel?

Yes, and that fit is exactly what an SDK gives you. The UI is fully customizable. You can theme it, restyle it, or drive the engine from your own interface in a headless setup. It acts as a native part of your product instead of a third-party tool bolted on. Swiss Post, which runs over a million personalized postcards a year on it, confirms this, calling CE.SDK “the only solution allowing a specialized, on-brand UI.”

We ship on web, iOS, and Android. Do we need three different tools?

No. One creative engine renders identically across web, iOS, and Android, so a template built on one platform looks the same on every other and your team keeps one mental model instead of maintaining three separate integrations. You build once instead of rebuilding the editor per OS.

Our users want AI features. Do we have to build those too?

AI editing ships inside the editor, not as a separate project. Background removal and generative tools like text-to-image and image-to-video are ready to use, and CE.SDK is bring-your-own-model, so you plug in the providers you want instead of running inference yourself. That can move the top line on its own.

We are seeing a 10x increase in new signups month over month in May — that was due to the fact that you can now auto-generate really beautiful-looking ads from scratch.

Hikari Senju — Founder & CEO, Omneky

Where does our users’ data actually go?

By default, processing happens client-side, in the browser or on the device, so images and video never touch our servers. For customers with strict data-security requirements that is often a deciding factor, and it is increasingly named as a must-have in AI-driven research. Nothing leaves the user’s machine unless you deliberately choose a server-side workflow. And since IMG.LY is based in Germany, GDPR compliance and EU data residency come built in.

If we build on this, are we locked in?

It is a fair concern, and a different one from whether the vendor sticks around. Your designs and templates stay yours. You can export both your scenes and the rendered output, as PNG, JPEG, PDF, and the scene file itself, and you can run the engine on your own infrastructure, fully headless if you want. You are integrating a component, not handing your data model to someone else, and switching the component out later is possible.

Can we depend on this provider when it matters?

IMG.LY is an established company based in Germany, with a real team of people behind the product rather than a side project that could disappear next quarter. 600+ organizations, from startups to large global enterprises and government agencies, run the SDK in production, together generating around 500 million creations a month. Names on that list include Mercedes-Benz, Lufthansa, SAP, Polaroid, and Ticketmaster. You also get a dedicated contact who stays reachable when you need help, not an anonymous ticket queue, and the SDK keeps shipping new capabilities release after release, so the foundation you build on only gets stronger.

How do we know it’ll work before we’re all in?

You don’t have to trust the pitch, you can prove it yourself. The trial is self-serve. A developer can spin up an account and start building with the real SDK without going through sales, then judge the integration and the docs firsthand. It doesn’t need to take long either, thanks to the AI developer tooling that helps your assistant generate accurate integration code. That’s how most teams who picked IMG.LY decided, because seeing it run in your own stack removes the guesswork before you commit.

Recognize a few of these?

These are exactly the questions a careful team should ask before building on someone else’s technology. If you’re working through them, we’re glad to help: try the SDK yourself or contact our team. Reaching out is no commitment, and we’re happy to dig into your specific case. And to see how it played out for other companies, our case studies offer interesting insights.

Starter Kits: A New Way to Use and Configure Our Editor

João — Wed, 22 Apr 2026 17:14:54 GMT

When we built the first version of CE.SDK’s mobile editors, we optimized for speed and ease of integration. We designed a set of ready-made solution editors: Photo, Design, Video, Postcard, and Apparel Editor. Each is built around the use cases we believed would meet the needs of most of our customers. With one component and a license key, you had a full creative editor running in your app. That approach got products to market fast, and for many teams it was exactly the right starting point.

But products evolve, and so do the needs around it.

As integrations matured, teams needed deeper configuration, more granular control, and the ability to shape the editor into something uniquely theirs. What starts as “give us a working photo editor” becomes “give us our photo editor.”

That’s why, starting with CE.SDK v1.73 on iOS and Android, CE.SDK evolves from a set of preconfigured solution editors into a single, flexible Editor foundation paired with Starter Kits: complete, working editor projects that you own, customize, and control entirely.

Here’s what that means and why it’s worth it.

What changed

Previously, you integrated CE.SDK by dropping in a solution-specific component: PhotoEditor, DesignEditor, VideoEditor, ApparelEditor and PostacardEditor. Each came with built-in UI, default behaviors, and preconfigured toolbars. You could adjust some of it through configuration APIs, but the foundation was managed internally by our SDK.

With v1.73, those components are replaced by a single Editor entry point that starts minimal without any configuration and a set of Starter Kits that provide the full experience you’re used to.

Each Starter Kit matches a previous solution: Photo, Design, Video, Postcard, and Apparel. You download it, drop it into your project, and you have the same editor as before.

With Starter Kits, the setup becomes explicit and lives in your own codebase. Instead of relying on built-in defaults inside a prebuilt solution, you start from a ready-made configuration structure that you can inspect, own, and change directly.

And if none of the starter kits match your needs, you can build your own editor from scratch using our configuration directly.

Before, you were always starting from one of the predefined solutions and trying to bend it to fit your use case, which was limited and often frustrating. Now, the building blocks are there for you to assemble however you want.

What are the benefits?

The full picture in one place

Configuration used to be spread across multiple APIs and documentation pages. It wasn’t always clear what you could customize, where to find the right setting, or whether a particular behavior was even exposed for modification.

With a starter kit, the configuration is the documentation. Open the configuration file, and you see every callback, every toolbar item, every setting, the complete scope of what the editor does.

This also makes production behavior safer. Critical parts of the editor setup are now explicit in your code rather than implied by SDK defaults. Your team can see exactly what is configured, why, and review changes through your normal development process.

More control and capability over your editor

Starter kits unlock customizations that were simply not possible before: restructure how the editor initializes; inject custom logic between loading phases; defines layouts from the ground up; build workflows that are unique to your product; and more.

The old configuration APIs drew a line between “customizable” and “not customizable.” Starter kits erase that line. If you can see it in the code, you can change it.

In simple terms, you can now adapt the editor to your product, instead of adapting your product to the editor, and this was one of our main drivers.

Opt in, never opt out

Until now, an SDK update could introduce visible changes to your editor, like a new toolbar button, a different menu option, a changed default, without you asking for it. Your release process had to account for UI changes that came from a dependency, not from your team.

With CE.SDK v1.73, that dynamic is reversed. SDK updates bring new features, engine improvements, performance gains, and bug fixes. But your editor UI stays exactly as you left it. New features and components become building blocks you adopt on your own schedule, documented, with examples, and ready when your implicit decide to add them. Nothing shows up in your product unless you put it there.

Two paths, your choice

Start from a Starter Kit
Download the one that matches your use case, run it, and customize from there. Each kit is a proven, production-ready structure that handles the complexity of editor setup so you don’t have to figure it out from scratch. Most teams will take this path.

Build from scratch
If you have very specific requirements or prefer to architect everything yourself, the Editor view accepts raw configuration directly. The starter kits serve as comprehensive, living reference implementations. Even if you never use them directly, they show you exactly how every piece fits together.

The engine you rely on hasn’t changed

It’s worth saying clearly: the core of CE.SDK, the rendering engine, export pipeline, asset handling, performance characteristics, and everything that powers the creative experience is unchanged. We improved how you configure and integrate the editor, not what powers it.

The Tradeoff

The old one-component integration was genuinely convenient for getting started. With starter kits, you have a small module in your project, a handful of well-organized files that you own and maintain.

We believe this is the right tradeoff for any team building a real product. The moment you need to customize anything you’ll be able to clearly structured and to modify without limits.

At its core, this change is about moving from implicit behavior to explicit control, and we believe that is the better long-term contract with our customers. Your editor should not change in meaningful ways unless you decide it should. Your upgrade path should be predictable. And your team should be able to understand the integration by reading its own code.

Getting Started

We’ve written detailed migration guides that walk through every change with before-and-after examples. Each guide also links to the starter kit repositories for your platform:

	iOS	Android
Migration Guide	Upgrading to v1.73	Upgrading to v1.73
Photo Editor	GitHub	GitHub
Design Editor	GitHub	GitHub
Video Editor	GitHub	GitHub
Postcard Editor	GitHub	GitHub
Apparel Editor	GitHub	GitHub

We’re here to help with the transition. If you have questions, reach out to our support team or open an issue on the starter kit repositories.

CE.SDK v1.69 Release Notes

Neslihan — Fri, 27 Feb 2026 18:29:53 GMT

v1.69 is the biggest developer experience release we’ve shipped. Agent Skills collapse the time from idea to working editor from days to under a minute. Production-ready Starter Kits eliminate days of boilerplate setup. PPTX and Canva Importer lets users bring their existing designs directly into your editor. And a comprehensive video overhaul across Web, iOS, and Android closes the gap between CE.SDK and native professional editing apps.

Let’s dive in!

Introducing IMG.LY Agent Skills for CE.SDK

The biggest change to CE.SDK’s developer experience since launch.

AI agents have changed how developers build. With Agent Skills, CE.SDK works natively with your AI agent.

Install the plugin for Claude Code or Cursor and your AI coding agent becomes a CE.SDK expert with bundled access to guides, API references, and starter kits across 10 web frameworks. No external services, no MCP servers, no context-switching.

What your developers can now do:

/cesdk:build — Describe a use case in plain language. The agent detects the framework, pulls the right starter kit, and scaffolds a working project autonomously.
/cesdk:explain — Ask any CE.SDK implementation question and get answers adapted to the specific framework and experience level.
/cesdk:docs-[framework] — Retrieve guides and API references directly from the IDE, without leaving the editor.

For your product, this means: developers prototype in minutes, not days. Non-technical team members can now build and extend functional editors independently. And onboarding new engineers to CE.SDK shrinks from a multi-day ramp to a single session.

→ Quick Start & Full Announcement
→ Explore Agent Skills Documentation

Launch in Minutes with Production-Ready Starter Kits for Web

Previously, getting CE.SDK running in a real product meant adapting complex demo projects: stripping features, adjusting architecture, and undoing assumptions baked into example code.

Starter Kits replace all of that with pre-configured, production-ready editor UIs.

Specialized kits for Photo, Video, and Design editors are scoped, clean, and immediately deployable. Each kit starts with a focused feature set. You explicitly opt in to what you need, so your UI stays performant and your users don’t see capabilities they can’t use.

For your product, this means: a functional, shippable editor in minutes rather than days of setup. Clean architecture from day one, without accumulating technical debt.

→ Explore Starter Kit Documentation

Let Users Bring Their PPTX & Canva Designs to CE.SDK

Some customers don’t start from scratch. They have existing creative assets locked inside PowerPoint and Canva, and until now, bringing those into CE.SDK required manual recreation.

The new PPTX Importer removes that barrier.

Users can import native PowerPoint files or export Canva designs as PPTX and bring them directly into your CE.SDK-powered editor, with layouts, elements, and structure intact.

→ Try the PPTX Import Demo
→ View PPTX npm

Professional-Grade Video Editing Across Platforms

This release expands CE.SDK’s video features across web & mobile platforms.

Control Video Speed (iOS & Android)

Users can now adjust the playback speed of video and audio clips via a dedicated Speed UI. This has been one of the most-requested mobile features and is now available on both platforms.
→ View Video Speed Documentation iOS
→ View Video Speed Documentation Android

Create Video Groups in the Timeline (Web)

Multiple clips can now be combined into a single manageable unit with synchronized timing, trimming, and a unified timeline representation.

This enables complex, multi-layer content production and speeds up the design process.

Video Timeline Now Grows Automatically (Web)

The timeline now automatically adjusts its height based on content. Manual track configuration is gone, and complex projects are easier to pick up for your new users.

Use a Clutter-Free iOS Timeline (iOS)

Non-video tracks now show a single thumbnail per clip, reducing clutter in multi-layer timelines on your iOS app.

Full Changelog

See all technical details, breaking changes, and performance improvements in the v1.69.0 Changelog.

Thank you for building with IMG.LY.

Building a CapCut-Like Video Editor with CE.SDK and AI

Jan — Thu, 26 Feb 2026 22:50:34 GMT

Build this with AI in minutes

This tutorial is a great way to understand how CE.SDK works under the hood. But if you just want to get to the result, you can build this entire editor using IMG.LY Agent Skills, no manual setup required.

Install the skills, then run:

/cesdk:build Build a CapCut-like video editor with a dark theme, multi-track timeline, AI video/image/audio generation, background removal, and MP4 export.

Want to understand a specific concept in depth? Use /cesdk:explain, for example:

/cesdk:explain How does the video timeline and block hierarchy work?

Introduction

CapCut has set the standard for accessible video editing. Its dark UI, intuitive timeline, and AI-powered features make it feel like a professional tool that anyone can use. But what if you could build something similar embedded directly in your own web application in minutes? Yes, minutes!

In this tutorial, we’ll walk through how we built a CapCut-like video editor using IMG.LY’s CreativeEditor SDK (CE.SDK) for React. The finished editor includes:

A multi-track timeline for video, audio, text, and captions
Trim, split, and join operations on video clips
A CapCut-inspired dark theme built with CSS custom properties
AI video generation (text-to-video, image-to-video) via Minimax, Kling, and Pixverse
AI image generation (text-to-image, image editing) via RecraftV3, IdeogramV3, and GPT Image
AI audio generation (text-to-speech, sound effects) via ElevenLabs
AI text generation (copywriting, translation) via Anthropic Claude
Background removal — client-side, powered by WebAssembly/WebGPU
MP4 export directly from the browser

The entire editor is a single React component with roughly 180 lines of JavaScript and 100 lines of CSS.

Architecture Overview

CE.SDK has two main layers:

CreativeEngine: The headless core that manages scenes, blocks, assets, and rendering. It handles the video timeline, playback, and export entirely client-side.
CreativeEditor UI: A pre-built, customizable UI layer that wraps the engine with a dock, inspector, timeline, canvas, and navigation bar.

Plugins extend both layers. The AiApps plugin, for example, registers AI providers with the engine and injects UI components (dock buttons, canvas menu items, generation panels) into the editor.

Step 1: Project Setup

We scaffolded a React project with Vite and installed CE.SDK alongside the AI plugin packages:

npm create vite@latest capcut-like-editor -- --template react
cd capcut-like-editor
npm install

# Core SDK
npm install @cesdk/cesdk-js

# AI plugins
npm install @imgly/plugin-ai-apps-web
npm install @imgly/plugin-ai-video-generation-web
npm install @imgly/plugin-ai-image-generation-web
npm install @imgly/plugin-ai-audio-generation-web
npm install @imgly/plugin-ai-text-generation-web

# Background removal (runs locally via WASM/WebGPU)
npm install @imgly/plugin-background-removal-web onnxruntime-web@1.21.0

Why so many packages?

CE.SDK follows a modular plugin architecture. Each AI capability is a separate package with its own provider modules. This means you only bundle what you use if you don’t need audio generation, don’t install @imgly/plugin-ai-audio-generation-web. The @imgly/plugin-ai-apps-web package is the unifying layer that brings them all together into a single dock panel.

Step 2: The React Component

CE.SDK provides a first-class React wrapper via @cesdk/cesdk-js/react. The <CreativeEditor> component handles mounting, initialization, and cleanup:

import CreativeEditor from '@cesdk/cesdk-js/react';

const config = {
  // license: 'YOUR_CESDK_LICENSE_KEY',
};

const init = async (cesdk) => {
  // All setup happens here — theme, assets, plugins, UI customization
};

export default function VideoEditor() {
  return (
    <CreativeEditor config={config} init={init} width="100vw" height="100vh" />
  );
}

The config object is passed to the engine at creation time. The init callback fires once the cesdk instance is ready. This is where all our customization lives.

Pitfall: Silent init errors. The <CreativeEditor> component swallows errors thrown inside init. If something fails, the editor loads but appears broken with no console output. Always wrap init in a try/catch:

Step 3: Creating the Video Scene

A video editor needs three things at startup: asset sources, a video scene, and a timeline.

// Load built-in asset libraries (stickers, shapes, filters, typefaces, etc.)
await cesdk.addDefaultAssetSources();

// Load demo content (sample videos, images, audio) + enable upload slots
await cesdk.addDemoAssetSources({
  sceneMode: 'Video',
  withUploadAssetSources: true,
});

// Create a video scene with timeline
await cesdk.createVideoScene();

createVideoScene() sets up the scene hierarchy that powers the timeline:

Scene
 └── Page (represents the video canvas — 1920x1080 by default)
      ├── Track (video track — holds video clips in sequence)
      ├── Track (overlay track — text, stickers, images)
      ├── CaptionTrack (subtitles synced to playback)
      └── Audio (background music, voiceover, sound effects)

Each element on the timeline is a block with timing properties: timeOffset (when it appears) and duration (how long it plays). The engine handles rendering each frame, compositing layers, and synchronizing audio.

Browser support

Video editing relies on modern web codecs (WebCodecs API), which are available in Chromium-based browsers (Chrome, Edge, Brave). Safari and Firefox support is limited.

Step 4: The CapCut Dark Theme

CapCut’s visual identity is defined by its deep charcoal backgrounds, teal accent colors, and subtle elevation layers. CE.SDK’s theming system maps perfectly to this through CSS custom properties.

Setting the base theme

cesdk.ui.setTheme('dark');

This activates CE.SDK’s built-in dark theme. But we want to go further. We need CapCut’s specific color palette.

Custom CSS overrides

CE.SDK scopes all its UI under .ubq-public with data-ubq-theme and data-ubq-scale attributes. We override the CSS custom properties to inject our palette:

.ubq-public[data-ubq-theme='dark'][data-ubq-scale='normal'],
.ubq-public[data-ubq-theme='dark'][data-ubq-scale='modern'] {
  /* Deep charcoal backgrounds — darker than CE.SDK's default dark */
  --ubq-canvas: hsl(220, 15%, 8%) !important;
  --ubq-elevation-1: hsl(220, 13%, 12%) !important;
  --ubq-elevation-2: hsl(220, 12%, 15%) !important;
  --ubq-elevation-3: hsl(220, 10%, 18%) !important;

  /* High-contrast white text */
  --ubq-foreground-default: hsla(0, 0%, 100%, 0.92) !important;
  --ubq-foreground-light: hsla(0, 0%, 100%, 0.55) !important;

  /* CapCut's signature teal/cyan accent */
  --ubq-interactive-accent-default: hsl(190, 85%, 48%) !important;
  --ubq-interactive-accent-hover: hsl(190, 85%, 42%) !important;
  --ubq-interactive-accent-pressed: hsl(190, 85%, 36%) !important;

  /* Subtle borders — barely visible, like CapCut */
  --ubq-border-default: hsla(0, 0%, 100%, 0.08) !important;
}

The key design decisions:

Property	Value	Why
`--ubq-canvas`	`hsl(220, 15%, 8%)`	Near-black with a slight blue tint — matches CapCut’s canvas area
`--ubq-elevation-1/2/3`	12% → 15% → 18% lightness	Subtle elevation steps create depth without harsh contrast
`--ubq-interactive-accent-*`	`hsl(190, 85%, 48%)`	CapCut’s teal — used for buttons, selections, and progress bars
`--ubq-border-default`	8% opacity white	Nearly invisible borders that only appear on close inspection

Responsive scale

We also configure responsive scaling so the editor adapts to touch devices:

cesdk.ui.setScale(({ containerWidth, isTouch }) => {
  if ((containerWidth && containerWidth < 768) || isTouch) {
    return 'large'; // Bigger touch targets on small/touch screens
  }
  return 'normal'; // Standard desktop sizing
});

Step 5: AI Plugin Integration

This is where the editor transforms from a basic video tool into something that feels like CapCut’s AI-powered experience. CE.SDK’s plugin system lets us add all AI capabilities through a single unified AiApps plugin.

The unified AiApps approach

Instead of registering each AI plugin separately, @imgly/plugin-ai-apps-web provides a single entry point:

import AiApps from '@imgly/plugin-ai-apps-web';
import FalAiVideo from '@imgly/plugin-ai-video-generation-web/fal-ai';
import FalAiImage from '@imgly/plugin-ai-image-generation-web/fal-ai';
import OpenAiImage from '@imgly/plugin-ai-image-generation-web/open-ai';
import Elevenlabs from '@imgly/plugin-ai-audio-generation-web/elevenlabs';
import Anthropic from '@imgly/plugin-ai-text-generation-web/anthropic';

await cesdk.addPlugin(
  AiApps({
    dryRun: true, // Simulate for development — no API calls
    providers: {
      text2text: Anthropic.AnthropicProvider({ proxyUrl: PROXY_URL }),
      text2image: [FalAiImage.RecraftV3({ proxyUrl }) /* ... */],
      image2image: [FalAiImage.GeminiFlashEdit({ proxyUrl }) /* ... */],
      text2video: [FalAiVideo.MinimaxVideo01Live({ proxyUrl }) /* ... */],
      image2video: [
        FalAiVideo.MinimaxVideo01LiveImageToVideo({ proxyUrl }) /* ... */,
      ],
      text2speech: Elevenlabs.ElevenMultilingualV2({ proxyUrl }),
      text2sound: Elevenlabs.ElevenSoundEffects({ proxyUrl }),
    },
  })
);

Provider categories explained

Category	What it does	Models we configured
`text2text`	AI copywriting — improve text, translate, change tone	Anthropic Claude
`text2image`	Generate images from text prompts	RecraftV3, IdeogramV3, GPT Image
`image2image`	Transform existing images with AI	Gemini Flash Edit, GPT Image
`text2video`	Generate video clips from text descriptions	Minimax Video, Kling Video, Pixverse
`image2video`	Animate a static image into video	Minimax Video, Kling Video
`text2speech`	Convert text to spoken audio with voice selection	ElevenLabs Multilingual V2
`text2sound`	Generate sound effects from text	ElevenLabs Sound Effects

When multiple providers are configured in an array (like text2image), the UI automatically shows a provider/model selection dropdown so users can choose which AI model to use.

The proxy server requirement

Every provider takes a proxyUrl parameter. This is critical for production:

Browser  →  Your Proxy Server  →  AI Provider (fal.ai, ElevenLabs, etc.)
                ↑
          Injects API keys
          server-side

Your API keys should never be in client-side code. The proxy server receives requests from CE.SDK, attaches your API key, and forwards to the AI provider. During development, dryRun: true simulates generation without any API calls.

Background removal

Background removal is a separate plugin because it runs entirely client-side, no proxy needed:

import BackgroundRemovalPlugin from '@imgly/plugin-background-removal-web';

await cesdk.addPlugin(
  BackgroundRemovalPlugin({
    ui: { locations: ['canvasMenu'] },
  })
);

This uses ONNX Runtime (WebAssembly + WebGPU) to run an AI segmentation model directly in the browser. The first run downloads ~40MB of model weights, which are then cached. Select any image on the canvas and click “Remove Background” in the context menu.

Step 6: UI Customization with the Component Order API

CE.SDK’s UI is built from five customizable areas: Dock, Inspector Bar, Canvas Menu, Navigation Bar, and Canvas Bar. The Component Order API lets us insert, remove, and reorder components in each area.

Adding the AI button to the dock

We want the AI Apps button to be the first thing users see in the dock:

cesdk.ui.insertOrderComponent(
  { in: 'ly.img.dock', position: 'start' },
  'ly.img.ai.apps.dock'
);

insertOrderComponent takes a location specifier and the component(s) to insert. position: 'start' puts it at the top of the dock. Alternative positions include 'end', a numeric index, or relative placement with before/after matchers.

When a user selects a text or image block and right-clicks, we want AI options available:

cesdk.ui.insertOrderComponent({ in: 'ly.img.canvas.menu', position: 'start' }, [
  'ly.img.ai.text.canvasMenu',
  'ly.img.ai.image.canvasMenu',
]);

Passing an array inserts multiple components at once.

We add an Export button with a real click handler that triggers MP4 export:

cesdk.ui.insertOrderComponent(
  { in: 'ly.img.navigation.bar', position: 'end' },
  {
    id: 'ly.img.action.navigationBar',
    key: 'export',
    label: 'Export',
    icon: '@imgly/Download',
    onClick: async () => {
      const engine = cesdk.engine;
      const page = engine.block.findByType('page')[0];
      if (page) {
        const blob = await engine.block.exportVideo(page, {
          mimeType: 'video/mp4',
        });
        const url = URL.createObjectURL(blob);
        const a = document.createElement('a');
        a.href = url;
        a.download = 'video.mp4';
        a.click();
        URL.revokeObjectURL(url);
      }
    },
  }
);

The exportVideo method renders every frame of the timeline (compositing video tracks, overlays, text, captions, and audio) into an MP4 blob entirely in the browser.

Step 7: Wiring Generated Audio into the Asset Library

AI-generated audio (speech and sound effects) is stored in provider-specific history sources. To make this audio browsable alongside regular audio assets, we inject the history source into the audio asset library entry:

const audioEntry = cesdk.ui.getAssetLibraryEntry('ly.img.audio');
if (audioEntry != null) {
  const existingSourceIds = Array.isArray(audioEntry.sourceIds)
    ? audioEntry.sourceIds
    : audioEntry.sourceIds({});
  cesdk.ui.updateAssetLibraryEntry('ly.img.audio', {
    sourceIds: [...existingSourceIds, 'ly.img.ai.audio-generation.history'],
  });
}

Now when users open the Audio panel in the dock, they’ll see their AI-generated audio alongside the sample library.

Step 8: Custom Labels with i18n

CE.SDK’s i18n system lets us customize any UI string. We used it to make the AI prompt fields more inviting:

cesdk.i18n.setTranslations({
  en: {
    'ly.img.plugin-ai-video-generation-web.fal-ai/minimax/video-01-live.property.prompt':
      'Describe your video...',
    'ly.img.plugin-ai-image-generation-web.fal-ai/recraft-v3.property.prompt':
      'Describe your image...',
  },
});

Translation keys follow the pattern {plugin-id}.{provider-id}.property.{field}. You can also add multi-language support by including keys for de, fr, es, etc.

The Complete File Structure

capcut-like-editor/
├── src/
│   ├── App.jsx              # Root — imports VideoEditor + theme CSS
│   ├── VideoEditor.jsx      # The entire editor (single component, ~180 lines)
│   ├── capcut-theme.css     # CapCut dark theme overrides (~100 lines)
│   ├── index.css            # Global reset (margin/padding/overflow)
│   └── main.jsx             # React entry point
├── index.html
├── package.json
└── vite.config.js

That’s it. The entire CapCut-like editor is three files: one React component, one CSS file, and a thin App wrapper.

What We Get Out of the Box

Because CE.SDK’s video UI is pre-built, we didn’t write any code for these features. They come from createVideoScene() and the default UI:

Multi-track timeline with drag-to-reorder, drag-to-resize
Trim and split — drag clip edges or use the split tool
Join and arrange — drag clips between tracks
Transform — crop, flip, rotate via the inspector
Playback controls — play, pause, seek, scrub
Text overlays — add styled text with the text tool
Stickers and graphics — from the asset library
Filters and effects — LUT-based color grading
Undo/redo — full history stack
Zoom and pan — standard canvas navigation

The AI plugins add their own UI components (dock buttons, generation panels, context menu items) through the plugin system. We just positioned them where we wanted.

Going to Production

To take this from a prototype to production, you need three things:

1. License key

Get a free trial key at https://img.ly/forms/contact-sales/ and set it in the config (yes, if you want to ship to production you’ll have to talk to our lovely colleagues in sales):

const config = {
  license: 'YOUR_CESDK_LICENSE_KEY',
};

2. Proxy server

Set up a server that forwards AI requests with your API keys:

const PROXY_URL = 'https://your-server.com/api/ai-proxy';

See the CE.SDK Proxy Server guide for Express.js and other server examples.

3. Disable dry run

Remove dryRun: true from the AiApps configuration to enable real AI generation.

Key Takeaways

CE.SDK does the heavy lifting. The timeline, playback, rendering, and export are all handled by the engine. We wrote zero video processing code.
The plugin system is powerful. Six AI capabilities were added with a single addPlugin(AiApps({ ... })) call. Background removal was one more call.
CSS custom properties make theming painless. We matched CapCut’s aesthetic by overriding ~25 CSS variables. No forking, no patching.
The Component Order API is the customization backbone. insertOrderComponent with position-based placement is the cleanest pattern for adding UI elements.
Wrap init in try/catch. CE.SDK’s <CreativeEditor> swallows errors silently. This is the single most important debugging tip.
Pin your package versions. All @imgly/* plugins must match the @cesdk/cesdk-js version exactly. A version mismatch (like 1.68 vs 1.69) will cause peer dependency conflicts.

Resources

Introducing IMG.LY Agent Skills

Neslihan — Mon, 16 Feb 2026 18:54:06 GMT

TL;DR
We are moving beyond static documentation. IMG.LY Agent Skills is a specialized intelligence layer that transforms AI coding assistants (Claude Code, Cursor, Windsurf) into CE.SDK implementation experts. Instead of mapping APIs manually, you now inject our full SDK expertise directly into your IDE and coding agent.

→ Explore Agent Skill Documentation
→ Access Agent Skills GitHub Repository

The “Quick Start” Paradox

In our recent analysis of client integrations, a consistent pattern emerged: even with a high-performance SDK, the “Time to First Edit” is often delayed by research. Developers spend a significant portion of their initial integration time context-switching between their IDE and documentation, manually mapping complex API hierarchies to their specific framework.

We figured: You don’t just need better documentation; you need an expert partner who is already inside your codebase.

The Solution: IMG.LY Agent Skills for Web

We are moving beyond static documentation. Today, we are launching IMG.LY Agent Skills, a specialized intelligence layer that transforms AI coding assistants like Claude Code, Cursor, and Gemini into CE.SDK implementation experts.

Instead of searching for answers, you now give your agent our complete SDK knowledge with a single command.

How It Works: Your Autonomous Implementation Partner

By injecting versioned, live documentation and framework-specific starter kits directly into your agent’s context window, we’ve created three adaptive paths to launch:

1. The Explain Path

Architecture is complex. The Explain Skill acts as a digital implementation partner that adapts to you. It can provide a technical deep-dive on the export pipeline for a Senior Dev or a high-level summary for a Product Manager, in any language and at any level of detail.

2. The Build Path

Command your agent to “Add a photo editor to my React app.” The agent autonomously detects your environment, pulls the correct starter kit, and handles the boilerplate. It doesn’t just show you code; it builds the foundation for you.

3. The Docs Path

Stop the tab-switching. Your agent retrieves versioned API references offline, keeping your focus entirely within the terminal or IDE.

The Shift to Autonomous Engineering

This release marks a strategic shift for IMG.LY. We believe the next era of software development isn’t about “better documentation,” but about Portable Expertise.

By providing that AI with a “skill” rather than a manual, we are empowering teams to move from a multi-month build cycle to a 30-second scaffold.

Supported Frameworks

The initial Web release supports 10 frameworks out of the box:
React · Vue.js · Svelte · Angular · Next.js · Nuxt.js · SvelteKit · Electron · Node.js · Vanilla JS

Get Started

Run the command:

npx skills add imgly/agent-skills

For example, Claude Code users, can run:

claude plugin marketplace add imgly/agent-skills

claude plugin install cesdk@imgly

→ Explore the full Agent Skill Documentation

The “editor of your dreams” is now a conversation away.

Join 3,000+ creative professionals who get early access to new features and updates. Subscribe.

From Prompt to Editor: Running CE.SDK Inside ChatGPT with the Apps SDK

Jan — Fri, 19 Dec 2025 15:43:16 GMT

With the new ChatGPT Apps SDK and Model Context Protocol (MCP), chat interfaces are starting to look less like Q&A tools and more like places where work actually happens. To explore what that means for creative workflows, we built a small technical demo: CE.SDK running directly inside ChatGPT.

From a user’s perspective, the flow is almost trivial. You ask ChatGPT for something like an ecommerce template. ChatGPT searches our template catalog, selects a matching design, and opens it instantly in a fully interactive CE.SDK editor, right inside the chat interface. What looks like a preview is, in fact, a real editor loaded with a real template scene.

This isn’t meant as a product announcement. It’s a technical proof of concept showing how creative SDKs can plug directly into AI-native interfaces.

CE.SDK as a ChatGPT App

The integration is built around a custom MCP server that exposes CE.SDK to ChatGPT as a tool. The server speaks OpenAI’s JSON-RPC–style MCP and implements the standard lifecycle methods (initialize, tools.list, tools.call, resources.read). It knows about our premium template catalog and emits structured payloads that the frontend understands.

On the client side, a Next.js app listens to tool output events streamed from ChatGPT, renders CE.SDK widgets, and hydrates them with the payloads returned by the tool, such as a scene URL, placeholder values, or export permissions. Templates are loaded via CE.SDK’s Template API, either from a URL or from a serialized scene string.

Under the hood, the stack is fairly conventional:

Next.js 15 (App Router)
CE.SDK Web / CreativeEngine
A custom MCP handler to normalize JSON-RPC
Vercel for hosting

What’s new is not the technology itself, but the context in which it runs.

Working with MCP in Practice

The hardest part of the demo wasn’t CE.SDK. It was MCP.

OpenAI’s MCP implementation is extremely strict. Even the smallest schema mismatch can trigger the infamous “TaskGroup 424” error, usually without any hint as to what went wrong. In many cases, the HTTP response is technically successful, but the JSON structure doesn’t match the expected schema closely enough.

The key lesson here is to treat MCP responses as hard contracts:

Validate every response against a schema (for example with zod).
Mirror OpenAI’s field names exactly, even for empty or optional capabilities.
Assume that a 424 almost always means “your JSON shape is wrong”.

Another important insight was how critical visual context is in chat-based tools. If your MCP responses don’t include thumbnails or preview images, ChatGPT will often fall back to rendering links. For creative tools, that immediately breaks the experience. In a chat UI, visuals aren’t an enhancement. They are the interface.

State handling also requires a shift in mindset. ChatGPT can replay tool calls, and each prompt effectively creates a new widget instance. You can’t rely on mutating an existing editor. The frontend needs to be idempotent: load scenes from serialized state first, then apply changes. Every tool call should be treated as a fresh render.

Why This Pattern Matters

This demo points to a broader change in how creative software may be accessed. Chat becomes a coordination layer, not just a conversational one. Instead of explaining how something could be designed, the AI opens the actual editor and lets the user continue from there.

For CE.SDK, this fits naturally. Editors become embeddable capabilities rather than standalone applications, and AI systems become the entry point into creative workflows. Prompting turns into doing.

Beyond OpenAI: The MCP UI Standard

Although this demo uses OpenAI’s MCP, the architecture maps cleanly to the new MCP UI standard recently introduced by Anthropic. That standard aims to make tool definitions and UI rendering more consistent across models and platforms.

Because this integration already separates tool logic from UI rendering and relies on structured, explicit payloads, transferring it to Anthropic’s MCP UI model is conceptually straightforward. CE.SDK can act as a reusable creative surface across ChatGPT, Claude, and future AI app ecosystems.

This demo is intentionally small and technical, but it highlights a meaningful shift: AI systems that don’t just describe creative outcomes, but open the tools to actually create them.

What is Visual Prompting?

Jan — Tue, 29 Jul 2025 13:06:11 GMT

A New Paradigm for Creative AI, Built by IMG.LY

To say it’s trite to refer to the impact of AI in this or that domain as disruptive or groundbreaking would be an understatement. Yet, few areas have been as profoundly affected as the creative process. With just a text prompt, anyone can produce stunning images, remix visual styles, and explore design possibilities at a scale and speed never seen before. AI has inserted itself so quickly into this process that its gone from curious novelty to an essential part of the creator toolchain.

The more serious adoption we see, however, the more key limitations of today’s AI tooling come into focus: the prompt itself.

Text alone, for all its expressive power, struggles to capture the essence of visual intent. Most creative work doesn’t begin with a sentence it begins with a sketch, a layout, a mood board, or an arrangement of elements. Visual ideas are shared by pointing, placing, showing.

At IMG.LY, we have begun to think about better ways to direct AI for visual generation, the term we use is Visual Prompting.

Visual Prompting: the practice of composing a visual scene or layout as input for a generative model.

Instead of describing what you want with paragraphs of text, you show it directly using a canvas of images, text, spatial cues, and annotations. This visual composition then becomes the prompt for the AI to generate new content in return. It’s a more natural, intuitive, and powerful way to collaborate with AI, especially when integrated directly into the creative process.

Problem: the Chat Disconnect

The current generation of AI tools has largely been shaped by language-first interfaces. Whether it’s ChatGPT for writing or Midjourney for image generation, the assumption is the same: the user will type a descriptive prompt, and the AI will generate a result based on it.

But when it comes to design, this workflow quickly runs into friction. Visual ideas are inherently spatial and non-linear. Trying to express layout, balance, mood, or specific spatial relationships through text can feel like trying to describe a painting over the phone. It’s possible but unnecessarily cumbersome.

A designer might want to:

Indicate that a certain area in the image should be blue.
Replace a background with a texture sample.
Position a character precisely in a composition.
Annotate which parts of a scene to preserve or modify.

All of these are difficult to express fluently in text. But they’re effortless in a visual interface. The truth is: an image is worth more than a thousand words when prompting an image.

What Is Visual Prompting?

Visual Prompting is a multimodal approach to generative AI, where the input to the model is not just text, but a full visual composition: images, text, annotations, and layout.

Rather than prompting AI in isolation, the user builds their intent on a canvas. This might include:

Reference images that communicate mood or style.
Text blocks indicating desired copy or instructions.
Annotations pointing to specific areas with notes like “make this glow” or “replace this object.”
Spatial composition: where elements are arranged meaningfully to convey intent.

The visual prompt is then interpreted by a multimodal model such as OpenAI’s gpt-image-1 to generate new visual content that reflects not only the textual description, but also the visual context.

How Visual Prompting Works in CE.SDK

About time for an example. As part of our recent AI released we demoed how to use OpenAIs gpt-image-1 model to build visual prompting into CreativeEditor SDK (CE.SDK).

Here’s what the process looks like inside CE.SDK:

Compose Visually: The user creates a layout with reference content, uploaded images, icons, color schemes, design elements, placeholder text, and annotations. This composition represents the “prompt” in visual form.
Add AI Layers: With a single click, the user can trigger image generation using CE.SDK’s built-in AI plugin. The plugin sends the visual context (alongside any optional text input) to a multimodal model capable of interpreting both.
Refine and Iterate: Users can adjust the layout, reposition elements, change annotations, or layer in new references, then prompt again. Because the canvas is interactive and editable, the feedback loop is tight.
Build Up Complexity: Over time, users can layer generated images with manually designed components or other generated outputs, creating rich compositions that blend AI creativity with human direction.

This workflow turns the traditional prompt/response cycle into a conversation between the designer and the model, with the canvas acting as the shared language.

Who Is Visual Prompting For?

The use cases for Visual Prompting extend across industries:

Creative teams can go from reference to generation in seconds, iterating visually instead of wrangling prompts.
Marketing teams can generate regionalized or personalized creative variants from a shared layout.
Product designers can prototype in context, turning layouts into realistic screens without leaving the editor.
Storytellers and content creators can use annotated sketches to generate detailed illustrations or scene variations.
E-commerce platforms can give sellers the power to visually customize their brand materials with AI assistance.

In every case, Visual Prompting replaces friction with flow and text-based prompting with something more expressive, more reliable, and more fun.

Built for This: Multimodal Models and CE.SDK’s Plugin System

Visual Prompting is only possible because of two parallel advancements:

Multimodal AI models, such as OpenAI’s gpt-image-1, that can interpret both images and text, understand spatial relationships, and respond to annotated cues.
A flexible, composable editor SDK like CE.SDK, which enables the construction of visual prompts on a live canvas, and makes it easy to integrate AI models directly into the design flow.

Our SDK was built from the ground up to support AI-first creative workflows. Its plugin architecture allows you to add any model or API, image generation, video generation, captioning, text rewriting and use it natively inside the editor without the need to switch tools or copy/paste.

Generative AI’s full potential is only unlocked when it is embedded directly into the tools creatives use not siloed in chatbots or separate interfaces. Visual Prompting allows that embedding to go even deeper, aligning the mode of input (visual) with the desired output (visual).

Explore It Yourself

🎨 Try out Visual Prompting in our AI Editor demo
📘 Learn How to Integrate AI into CE.SDK
💬 Contact Us to Bring Visual Prompting to Your Product

Build vs. Buy: Is Fabric.js Right for You

Jan — Thu, 15 May 2025 09:31:43 GMT

When you’re building a design editor into your product whether for social content, marketing assets, video, or design workflows choosing the right foundation matters. Many teams start with Fabric.js, an open-source canvas library, attracted by its flexibility and permissive license. But as the project scope grows, so do the limitations.

This article explores the advantages, but also the pitfalls of building a creative tool with Fabric.js and why many product teams ultimately decide to switch or start with a commercial SDK like IMG.LY’s CreativeEditor SDK (CE.SDK).

Consult our comparison page of Fabric.js and IMG.LY for a feature by feature breakdown of the differences between our SDKs and Fabric.js.

Why Teams Switch from Fabric.js

We’ve spoken with dozens of companies, from startups to enterprise clients, who evaluated Fabric.js and in many cases opted for IMG.LY instead. Here’s what we consistently hear:

1. Time-to-Market vs. Reinventing the Wheel

Open-source is attractive at first. But once the excitement of building wears off, teams often face a sobering reality: stitching together a decent editing experience from Fabric.js is a months-long effort.

“We used Fabric.js before, but if IMG.LY’s SDK is cost-effective, we’d rather not reinvent the wheel.”

— B2B SaaS Prospect

They want a modern editor UI, fast iteration cycles, and a stable base not to spend their next quarter building the basics like zoom, text editing, object grouping, layer management, or responsive templates.

2. Lack of Advanced Features

Fabric.js excels at low-level canvas manipulation, offering a robust API for working directly with objects on the HTML5 canvas - shapes, images, text, and transformations. It’s a great starting point for developers who want fine-grained control over rendering and interactivity. However, it falls short when it comes to modern editing paradigms. Fabric.js provides no native concept of reusable templates, lacks support for layout and design constraints like alignment guides or snapping behavior, and does not include higher-level abstractions for content-aware or AI-powered editing. In short, Fabric.js gives you a raw toolkit, not a polished, plug-and-play editing solution. If you’re building a sophisticated design application, you’ll need to layer these capabilities yourself or integrate additional frameworks to bridge the gap.

“We needed a template system and advanced photo editing. Open-source libraries couldn’t do it out of the box.”

— Marketplace Platform Prospect

Teams chasing parity with Canva or Adobe tools quickly hit walls. What starts as a proof-of-concept becomes a rebuild of the tables stakes of design editing.

Real-World Use Cases & Vertical-Specific Challenges

We’ve seen teams from various industries reach the same conclusion: Fabric.js doesn’t scale to meet their creative, technical, or business needs. A few common themes:

E-commerce & Web-to-Print: Retailers building product customization flows need template constraints, consistent output quality, and export formats that go beyond canvas. They can’t afford rendering inconsistencies or unpredictable export fidelity.
MarTech & Social Media Tools: Marketers need batch generation, AI-assisted creative workflow, brand constraints and consistent creative output across devices. Fabric.js lacks built-in support for these workflows.
Mobile-first Apps: Teams building hybrid apps or web apps that need to be functional on mobile struggle with Fabric.js’ lacklustre mobile support. The inability to deliver a consistent UX across iOS, Android, and Web leads to dropped features or split tech stacks.
Video and Multimedia Platforms: Fabric.js has no native video support, track-based editing, or multi-frame logic. Prospects in this space frequently abandon their Fabric.js POCs once real constraints emerge.

A major pain point across all verticals is rendering consistency. CE.SDK renders identically across browser, server, and mobile thanks to a shared rendering core (CreativeEngine). This is critical for:

Workflows that begin on web and finish on mobile
Server-side generation (e.g., previews, PDFs, batch exports)
Feature parity and UX reliability across platforms

Check out our demo page to explore these use cases.

Technical Debt: The Hidden Cost of Fabric.js

On paper, Fabric.js seems like a fast way to get started. But its real cost emerges over time in performance bottlenecks, missing architecture, and maintenance drag.

Aging Codebase and Patchy Maintenance

As of 2025, Fabric.js has over 400 open issues on GitHub, some dating back years. Many feature requests such as better performance for large object sets, async rendering, or text-on-path improvements are either unresolved or uncertainly prioritized.

The core maintainers do their best, but progress is slow. You’re relying on volunteer effort for critical infrastructure.

Slow Issue Resolution

Many of the GitHub issues (e.g., #5885, #6582) highlight long-standing rendering bugs and performance problems. These are hard to fix, and updates can break your own hacks or workarounds.

Unlike a commercial SDK, Fabric.js doesn’t guarantee backward compatibility. No SLAs. Little roadmap visibility.

Customer Feedback: Why They Walked Away

Here are some soundbites from our customers and prospects on why they decided against building with Fabric.js:

“The quality and UX didn’t meet our standards.”
Product Customizer

“Developer experience was poor. We spent more time debugging than building features.”
B2B SaaS Ad Design

“We needed enterprise support, scalability, and documentation. Open source didn’t cut it.”
Web to print Customer

“Cross-platform support and minimum SDK version constraints were a major blocker.”
Claim Management Application

Voice of the Dev

Beyond enterprise evaluations, developers working with Fabric.js often share their struggles publicly on GitHub, Stack Overflow, and developer forums. Their commentary offers real-world insight into day-to-day frustrations:

“FabricJS doesn’t seem to work well on Mobile.”

Github issue #6980

“Fabric.js object controls don’t work until after a common selection is made. Had to hack around it with extra event listeners.”

StackOverflow

“Trying to integrate Fabric.js with Next.js + Rollup is a mess. Unexpected tokens, config rewrites — it doesn’t play well with modern bundlers.”

GitHub Issue #8444

“We’re blocked by the use of ‘unsafe-eval’ due to our content security policy. Fabric.js needs a rewrite to be CSP-compliant.”

GitHub Issue #9666

Concrete takeaways from developer feedback:

Basic UI control quirks: Common issues with selection handling and control behavior require manual workarounds.
Compatibility friction: Fabric.js does not play well with modern build tools like Next.js and Rollup without additional configuration or patching.
Security blockers: The reliance on unsafe-eval creates CSP conflicts, making it unsuitable for projects with strict security requirements.
Uncertain roadmap: Critical features like async rendering may sit open for years, and niche features important to your use case may never land if they don’t align with community priorities.

While every open source project faces these difficulties, they do represent systemic issues with performance, extensibility, and production-readiness. For teams building long-lived, customer-facing tools, this developer feedback is often a leading indicator of future roadblocks.

Back-of-the-Envelope Cost of Building with Fabric.js

Many teams underestimate the cost of building a design editor from scratch. Here’s a rough breakdown of the time investment we’ve heard from teams who tried:

Task	Estimated Engineering Time
Core canvas-based editor UI (zoom, drag, resize, selection, tool switching)	6–8 weeks
SVG support, snapping, grouping, and layer management	4–6 weeks
Template constraints	3–5 weeks
Cross-platform adaptation (ironing out issues on mobile browsers, defining fallbacks)	4–6 weeks
Bug triage, ongoing maintenance, and refactors (first 12–18 months)	6–9 weeks

Total: ~6–8 months of senior dev time just to get to feature parity with what CE.SDK offers out of the box.

This doesn’t include the cost of QA, product management, or future extensibility. Nor the opportunity cost of what your team could be building instead.

IMG.LY’s CE.SDK: A Ready-Made Solution Built for Growth

By contrast, IMG.LY offers a production-grade SDK built for cross-platform creative tools, backed by a dedicated engineering team and used by major apps across industries.

Category	IMG.LY (CE.SDK)	Fabric.js	Notes
Out-of-the-box UI	✅ Prebuilt modern UI	❌ Build from scratch	CE.SDK ships with full UI/UX patterns
Plugin System	✅ Native plugin architecture	⚠️ Manual code extension	IMG.LY supports formal plugin APIs
Free Drawing Tools	⚠️ Requires integration	✅ Built-in pencil + shapes	Fabric.js excels at shape-level control
Advanced Editing	✅ Vector + raster support, layering, filters	⚠️ Basic vector only	CE.SDK includes pro-grade editing capabilities
Templating System	✅ Dynamic placeholders + constraints	❌ Manual implementation	IMG.LY supports automation workflows
Video Editing	✅ Multi-track timeline editor	❌ Not supported	No video support in Fabric.js
Cross-Platform	✅ Web, iOS, Android, Node, Flutter	⚠️ Web only	IMG.LY has native SDKs and server tools
Asset Integrations	✅ Unsplash, Getty, Brandfolder	❌ Manual setup	CE.SDK integrates assets out-of-the-box
Enterprise Support	✅ SLAs, onboarding, deployment help	❌ Community support only	IMG.LY offers dedicated engineering help
Design File Import Support	✅ PSD, AI, INDD	⚠️ None	CE.SDK supports importing common design file formats
AI Editing	✅ Background removal, integrate any AI model for image/video/text/audio gen	❌ Not supported	AI-native editing pipeline available
Rendering Consistency	✅ Unified engine across platforms	⚠️ Browser dependent	CE.SDK ensures pixel-parity on all targets

When Does Fabric.js Still Make Sense?

This article might read like a put-down, but it’s not supposed to be that. Until now we were simply evaluating Fabric.js with respect to building fully-featured design editing tools. There are, however, a number of well-scoped use cases where Fabric.js excels. These include:

Building an internal tool, proof of concept, or educational app
Creating a custom, single-platform sketching or annotation layer
Developing a low-fidelity prototype with full code-level control
Building canvas collaboration tools such as drag and drop editors or interactive canvases, such as Miro.

An example of where Fabric.js worked well is a tool such as Mockola, a drag-and-drop diagram editor, where shape manipulation, lightweight rendering, and canvas-level control are more important than cross-platform design fidelity. Mockola’s is a good use case, because its emphasis on real-time collaboration and planning rather than high-fidelity design. This means that features like free drawing, drag-and-drop, and JSON-based serialization are a great fit, all of which Fabric.js supports out of the box.

For teams willing to invest in building and maintaining their own editor infrastructure and whose use case does not require cross-platform parity, advanced media editing, or enterprise scaling Fabric.js continues to be a compelling foundation.

However, for products that need to ship quickly, scale reliably, and support modern creative workflows across platforms, IMG.LY’s CE.SDK offers the infrastructure and polish that customers now expect.

The Bottom Line

As MIT CIO Symposium speaker Mark Holst-Knudsen put it:

“You shouldn’t build anything that’s available off the shelf because it’s not a source of competitive advantage if everybody else can avail themselves of it.”

For most companies, building an editor from scratch is not a competitive edge, it’s a distraction from shipping and scaling. And Fabric.js, while capable in the hands of specialists, isn’t a shortcut to the finish line.

IMG.LY’s CE.SDK gives you a head start with advanced editing features, AI automation, template support, plugin extensibility, rendering consistency, and battle-tested scalability across platforms. We put together a data report based on 600+ customers, 28 structured customer interviews, and a self-reported customer survey that speaks for itself when it comes to real-life outcomes.

So before you dive into Fabric.js and start reinventing core functionality, ask yourself: What competitive advantage are you going to build on top of a design editor and which functionality are you better served buying from a trusted vendor?

CE.SDK v1.49 Release Notes

Neslihan — Thu, 17 Apr 2025 12:55:42 GMT

Get ready to add more motion and emotion to your designs! From fully integrated GIF Support to the cinematic Ken Burns animation and polished text background on iOS, this update packs features that elevate storytelling and enhance visual impact. Let’s dive in!

Use GIFs in Your Designs on Web and Mobile

GIFs are now fully integrated into the CE.SDK UI. Add and manage animated images like any other media asset. Drop a GIF into your timeline to play it once, or create a loop by extending its duration. You’re in control.

This update delivers a consistent experience across Web and Mobile, empowering dynamic content creation for social media, marketing, and beyond.

Getting Started:
GIF support itself is built-in; no setup required. However, to add looping, enable the looping option in your assets. Find platform-specific guidance in our docs:

Add A Cinematic Ken Burns Animation

Give your images a cinematic touch with the new crop-zoom animation in CE.SDK. Inspired by the Ken Burns effect, this feature lets you smoothly pan and zoom across images, creating dynamic storytelling without complex editing.

Apply the Ken Burns animation directly from the CE.SDK animation library on the web. Learn how to use animations and their properties with our documentation.

Add Text Backgrounds on iOS

Make your text pop! Apply solid color backgrounds to text on iOS. Additionally, adjust the padding and rounded corners of your text background.

This feature ensures readability and adds a polished, professional touch to your designs. By elevating text design, your app helps users create visually compelling content that stands out. Available for Web and iOS, soon on Android!

Get started with our text background documentation for iOS.

Improvements

Speed Up Your HDR Video Editing

We’ve dramatically accelerated HDR-to-SDR video conversion in CE.SDK on iOS. While HDR videos previously required lengthy processing, especially for 4K content, they now convert to SDR with minimal delay, letting users start editing their footage faster than ever before.

Stay tuned: Smooth HDR-to-SDR conversion will also be available for Web with CE.SDK v1.50.

Upcoming

Boost Engagement and Accessibility with Captions

Captions are essential in today’s social media landscape, where videos often autoplay on mute and clear messaging is key. Our upcoming Captions for Web feature will allow you to:

Manually add subtitles to videos or upload an SRT file for seamless integration.
Adjust timing and formatting in an intuitive interface to match your brand’s style.
Embed captions directly into videos, ensuring they remain visible across all platforms.

Thanks for reading! View all bug fixes and changes in the Changelog.

3,000+ creative professionals gain early access to new features and updates. Don’t miss out, and subscribe to our newsletter.

CE.SDK v1.48 Release Notes

Neslihan — Tue, 08 Apr 2025 12:13:35 GMT

This release makes it easier than ever to tailor the Android editing experience to your brand. Customize the navigation bar to guide your users seamlessly and create an interface that feels uniquely yours.

Let’s jump in:

You now have more control over customizing the navigation bar on Android, ensuring that it reflects your brand and optimizes user flows.

Key Features:

Add Custom Buttons: You can add custom buttons to the navigation bar, to give users quick access to frequently used actions.
Rearrange Buttons: Arrange buttons in your navigation bar.
Integrate Custom Components: Customize the navigation bar by integrating unique elements that fit your app’s design and functionality.

These updates allow you to fully tailor the mobile editing experience to your specific needs.

💡 Did you know? In our previous update, we introduced customizability for the dock and inspector bar on Android, laying the foundation for more UI personalization.

We’re continuing that momentum by giving you full control over the navigation bar, ensuring that every part of your mobile editing experience can align with your product’s workflow.

Start customizing your navigation bar with our documentation.

Upcoming

Enjoy Vibrant HDR Videos
CE.SDK will support loading HDR videos on iOS, converting them to SDR on the fly for accurate and vibrant color representation, without the need for time-consuming transcoding.

Previously, HDR videos needed to be transcoded with delays, especially with large 4K videos. Soon, HDR content is processed instantly. Videos appear with correct color fidelity right away, making the editing experience more fluid and visually accurate for users.

This enhancement will remove friction in the editing workflow and lets your users start working with their content immediately. By offering a smoother, higher-quality experience, your app can boost user engagement and retention.

View all bug fixes and changes in the Changelog. Thanks for reading!

3,000+ creative professionals gain early access to new features and updates. Don’t miss out, and subscribe to our newsletter.

CE.SDK v1.46 Release Notes

Neslihan — Tue, 11 Mar 2025 15:58:30 GMT

Welcome to a new release! CE.SDK v1.46 introduces a more flexible and intuitive user interface for iOS, allowing you to optimize your workflow with a customizable dock and inspector. Read on for upcoming features.

Shape Your iOS Interface Around Your Workflow

We unlock full dock and inspector bar customization, giving you the flexibility to optimize your workflow. With an open API, you can now tailor these UI elements to better suit your editing needs, whether by reordering actions, adding custom functionality, or simplifying the interface.

Take control of your interface by modifying both the dock and inspector bar:

Rearrange & Customize Actions – Prioritize your most-used tools by reordering or removing actions to create a cleaner, more efficient workspace.
Add Custom Buttons & Views – Extend functionality by integrating custom buttons or views that trigger specific actions, such as opening an asset library or displaying additional information.

These enhancements provide a seamless, intuitive editing experience that adapts to your unique workflow needs.

Start customizing your dock and inspector bar today with our documentation.

Upcoming

Boost Engagement and Accessibility with Video Captions
Captions are essential in today’s social media landscape, where videos often autoplay on mute and clear messaging is key. Our upcoming Video Captions for Web will allow you to:

Manually add subtitles to videos or upload an SRT file.
Adjust timing and formatting in an intuitive interface to match your brand’s style.
Embed captions directly into videos, ensuring they remain visible across all platforms.

Stay tuned. Captions are coming soon!

View all bug fixes and changes in the Changelog. Thanks for reading!

3,000+ creative professionals gain early access to new features and updates. Don’t miss out, and subscribe to our newsletter.

How to Build a Short Video Generator Using CE.SDK

Eray — Tue, 04 Mar 2025 12:57:47 GMT

In the following, I’m presenting a simple cookbook for building an AI-based video generator app, as described in my previous blog post. We’re using a combination of different APIs to generate text, audio, and images and will compose & render the final video using the headless CreativeEditor SDK. We also call it the Creative Engine.

This cookbook showcases the powerful capabilities of our client-side Creative Engine. The engine enables real-time video generation directly in the browser, eliminating the need for server-side processing. What sets this approach apart is its ability to produce editable source files that can then be opened with CreativeEditor SDK.

This approach is giving users complete control over every aspect of your video–from text and images to animations and overall composition. This means your users can refine and perfect your content even after the initial generation.

Get the complete code on GitHub.

Scope

This tutorial focuses on building an app with a simple UX:

Input your keywords/topics
Choose between landscape or portrait format
Generate and preview your video
Edit the video in the CE.SDK video editor

The app flow we will create:

The post-editing we will get with CE.SDK:

Technical Overview

The app follows three major steps to generate the video.

A script is generated based on User input, the output is a structured XML file.
The XML script is parsed to extract text and image information. The extracted data will then be used to generate audio & image files through third-party APIs
All assets are loaded into the creative engine. This is where the composition, including animation and effects, takes place. The Creative Engine then exports a video and scene file, which can be edited with the Creative Editor.

Setup

We’ll use a boilerplate with Next.js, React, Typescript & Tailwind. Make sure you retrieve all necessary keys:

Anthropic (LLM)
ElevenLabs (text to speech)
fal.ai (text to image)
IMG.LY CE.SDK – Retrieve a free trial key

// Required environment variables
NEXT_PUBLIC_ANTHROPIC_API_KEY = your_claude_api_key;
NEXT_PUBLIC_FAL_API_KEY = your_fal_ai_key;
NEXT_PUBLIC_ELEVEN_LABS_KEY = your_eleven_labs_key;
NEXT_PUBLIC_IMG_LY_KEY = your_img_ly_key;

Implementation

1. Generate The Script

In this step, we’ll focus on generating the initial prompt and then passing it to the Anthropic API.

As with many things with LLM, there are many different strategies for structuring the initial prompt. From experience, the best result comes from providing examples of the desired output. We’ve decided to use an XML document; this can be easily parsed later on and is less error-prone compared to a JSON.

We now define the structure of how information should be saved in the XML.

<video>
  <group part="intro">
    <element>
      <text voiceId="50YSQEDPA2vlOxhCseP4" style="0.2">
        Did you know these fascinating facts about pyramids?
      </text>
      <image>Ancient Egyptian pyramid at sunset</image>
    </element>
  </group>
  <group part="content">
    <element>
      <text voiceId="50YSQEDPA2vlOxhCseP4" style="0.2">
        The Great Pyramid was the tallest structure for over 3,800 years!
      </text>
      <image>Great Pyramid comparison to modern buildings</image>
    </element>
  </group>
  <group part="outro">
    <element>
      <text voiceId="50YSQEDPA2vlOxhCseP4" style="0.4">
        The pyramids continue to reveal their secrets to this day...
      </text>
      <image>A giant 3D question mark hovering over the pyramids</image>
    </element>
    <element>
      <text voiceId="50YSQEDPA2vlOxhCseP4" style="0.4">
        Stay curious - there's always more to discover!
      </text>
      <image>Pyramids under starry night sky</image>
    </element>
  </group>
</video>

In this tutorial, we’ll focus on the format trivia only as shown in this example. For later iterations, however, I’m planning to implement different content formats (e.g., trivia, quiz, recipe, etc.). Each of these formats will have its example XML. Therefore, I’m nesting the XML in a simple format object to scale this up easily later.

interface Format {
  name: string;
  example: string;
}

const formats: Record<string, Format> = {
  trivia: {
    name: 'Trivia',
    example: `<video><group>...</group></video>` // Add example from above
  }
};

Using this format object with the example, we can now generate the prompt.

What do we need for this prompt?

Description of the task
Description of the desired output, incl. an example for the specified format
Topic as provided by the user

The topic provided by the user is passed to the function as a string.

export const createVideoScriptPrompt = (
  topic: string,
  formatName: string = 'trivia'
) => {
  const format = formats[formatName];
  if (!format) throw new Error(`Format ${formatName} not found`);

  return `
Format: ${format.name}
Topic: ${topic}

Please write a detailed script for this short video, considering the specified format and topic.
Include an introduction, main content sections, and an outro. Each section should have an image.
Structure the script as an XML Document with clear sections, descriptions for the images.
The image description should be written as a prompt. This prompt will be used to generate an image.
Put the description between the image tags. The video shouldn't be longer than 30 seconds.

Example format:
${format.example}`;
};

2. Generate All Assets

In the second step, we’ll parse through the LLM response, which should be the XML. We’ll create a simple parsing function to extract all text information that should be sent to text-to-speech and text-to-image AIs.

Please note that all these steps can be easily streamlined by using AI-assisted coding. Just provide the example XML as input and your desired output.

API Calls
When finding text & image tags in the XML, we’ll call API functions for text-to-speech and text-to-image. For this example, I’m using ElevenLabs & fal APIs. You will find all API calls in the api.ts.

Since the LLM generated a script that includes image prompts, make sure to pass them to the API.

export async function generateImage(prompt: string): Promise<string | null> {
  try {
    console.log('Generating image for prompt:', prompt);
    const result = await fal.subscribe('fal-ai/flux/dev', {
      input: {
        prompt: prompt,
        size: 'portrait_16_9',
      },
    });
    const typedResult = result as { images: { url: string }[] };
    console.log('Image generation successful. URL:', typedResult.images[0].url);
    return typedResult.images[0].url;
  } catch (error) {
    console.error('Error generating image:', error);
    return null;
  }
}

Timestamps
Last but not least, we need to come up with timestamps. What’s the duration of each segment? This is critical information for composing the video. Luckily, this is quite easy: Each scene is as long as the generated audio for each segment. This duration for the audio segments can be calculated: Most TTS like ElevenLabs provide timestamps along the audio file. These are typically character-based timestamps, so we first have to calculate the timestamps for each word and then the duration for the entire text section.

Ready For The Next Steps
All Asset URLs that are generated will be saved in a VideoBlock object for convenience. The duration of the VideoBlock is the duration of the audio, as calculated above.

interface VideoBlock {
  text: string;
  imageUrl: string | null;
  audioUrl: string | null;
  startTime: number;
  duration: number;
  wordTimestamps: Array<{ word: string; start: number; duration: number }>;
}

3. Generate The Video

We have everything together now: The completed XML with timestamps, duration, and all assets. It’s now time to generate the video using the creative engine.

Let’s first add an empty container in our HTML that will be referenced for initiating the creative engine.

{
  /* Add container for Creative Engine */
}
<div id="cesdk_container" className="invisible mt-8 rounded-lg bg-gray-100" />;

We can now initialize the engine. Use this code snippet from our documentation.

We’ll then set up a function that creates a simple composition using the provided VideoBlocks. The engine requires you to first create a scene, append a page to the scene, and then create tracks within the page. The tracks are basically the layers in the timeline. I recommend setting one track as a background track using the following snippet:

// Set video track as a background track by connecting the page duration to the video track
engine.block.setAlwaysOnBottom(videotrack, true);
engine.block.setPageDurationSource(page, videotrack);

The Creative Engine provides powerful API calls to style & manipulate blocks in many ways. Here is an example of how we can animate the images with a slow zoom effect:

const imageZoomAnimation = engine.block.createAnimation('crop_zoom');
engine.block.setInAnimation(image, imageZoomAnimation);
engine.block.setDuration(imageZoomAnimation, block.duration);
engine.block.setBool(imageZoomAnimation, 'animation/crop_zoom/fade', false);

Export The Video & Scene
Exporting the video is easy. Just pass the page to the export function. In our example, we’re also saving the scene file so we can edit the video later.

// Export video
const progressCallback = (
  renderedFrames: number,
  encodedFrames: number,
  totalFrames: number
) => {
  console.log(`Progress: ${Math.round((encodedFrames / totalFrames) * 100)}%`);
};

const blob = await engine.block.exportVideo(
  page,
  'video/mp4',
  progressCallback,
  {}
);

// Save scene to string
const sceneData = await engine.scene.saveToString();

// Create scene blob
const sceneBlob = new Blob([sceneData], {
  type: 'text/plain',
});

4. Add A Video Editor

The last step is to add the video editor for post editing and pass the scene file. With CE.SDK, this effort is reduced to adding a few lines of code. In the init function, we’re configuring the editor and adding callbacks for the export:

const initEditor = async () => {
        const config = {
          license: 'A-O53TWXK5bfyconUx7e53S5YU7DzjuGpMAH5vvKjLd0zBa6IhsoF7zChy1uCVbj',
          userId: 'guides-user',
          theme: 'dark',
          baseURL: '<https://cdn.img.ly/packages/imgly/cesdk-js/1.44.0/assets>',
          role: 'Creator',
          ui: {
            elements: {
              view: 'default',
              panels: {
              },
              navigation: {
                position: 'top',
                action: {
                  save: true,
                  load: true,
                  close: true,
                  download: true,
                  export: true
                }
              },
              dock: {
                iconSize: 'normal', // 'large' or 'normal'
                hideLabels: true // false or true
              }
            }
          },
          callbacks: {
            onUpload: 'local',
            onSave: (scene: string) => {
              const element = document.createElement('a')
              const base64Data = btoa(unescape(encodeURIComponent(scene)))
              element.setAttribute(
                'href',
                `data:application/octet-stream;base64,${base64Data}`
              )
              element.setAttribute(
                'download',
                `video-${new Date().toISOString()}.scene`
              )
              element.style.display = 'none'
              document.body.appendChild(element)
              element.click()
              document.body.removeChild(element)
            },
            onClose: () => {
              onClose();
            },
            onLoad: 'upload',
            onDownload: 'download',
            onExport: 'download'
          }
        }

Conclusion

By following this cookbook, you can streamline the process of AI-generated video creation, making it fast and efficient. This method is especially useful for content creators, educators, and marketers looking to automate video production while maintaining creative control.
Next, try experimenting with video styles, refining AI scripts, or exploring advanced editing.
Feel free to GitHub repo and share your creations with us on X. Happy creating!

3,000+ creative professionals gain early access to new features and updates. Don’t miss out, and subscribe to our newsletter.

CE.SDK v1.45 Release Notes

Neslihan — Tue, 25 Feb 2025 14:11:57 GMT

Welcome to CE.SDK v1.45! We’re expanding creative possibilities and refining the editing experience. Reactions are now available on Android, enabling users to record, edit, and personalize real-time responses.

We also introduce improvements like smoother development with TypeScript and better text rendering across languages.

These solutions help creators and developers work smarter, engage better, and deliver high-quality content.

Record Reactions to Videos on Android

Now available on Android, the popular Reactions feature lets users record and add their real-time responses to videos.

This is a must-have for engagement-driven content, mirroring interactions on platforms like TikTok and Instagram. Reactions have become a key element of personal branding and connecting with other creators on social media. With Reactions, users can:

Connect with their audience, duet, respond to trends, or join conversations.
Add a personal touch, react to fan art, viral moments, or inspiring videos.
Show thought leadership, comment on user questions, industry news, or niche topics.
Increase reach, boost visibility and interaction.
Monetize content, sell products and showcase collaborations.

Reactions make content creation fast, easy, and personal.

With our shipped UI, users can select a vertical or horizontal layout to record their reaction video, then customize it with text, filters, adjustments, and more.

It’s easy to crop, trim, and further adjust your video clip, providing a flexible starting point for seamless editing.

Try the live demo and enable Reactions for Android with our documentation.

Improvements

Ensure Text Rendering Across Languages

When a glyph is unavailable in the selected font, CE.SDK automatically falls back to a configurable default font. This prevents localization issues and ensures seamless text rendering, perfect for your global projects.

Simplify Development with Complete TypeScript Support

We’ve improved TypeScript support by exposing previously missing types used in our UI APIs. This enhancement, based on user feedback, reduces complexity and ensures smoother development workflows.

View the Changelog. Thanks for reading!

3,000+ creative professionals gain early access to new features and updates. Don’t miss out, and subscribe to our newsletter.

CE.SDK v1.44 Release Notes

Neslihan — Thu, 06 Feb 2025 07:31:00 GMT

Welcome to CE.SDK v1.44! This release brings a range of updates designed to enhance your editing workflow across platforms. Enjoy more vibrant and true-to-life visuals with expanded P3 color support on iOS and Android. On the web, streamline your editing process with the new ability to add multiple clips per track, making timeline management more efficient.

Let’s dive into the updates!

Simplify Your Video Editing Workflow

Add multiple clips to a single track on your video timeline! With CE.SDK video editing for the web, you can insert video clips, text, overlays, and more on one track, resulting in a more organized and efficient workspace.

This update empowers you to create richer, more dynamic sequences while keeping your timeline neat and easy to navigate.

Maximize Efficiency
Simplicity and speed are key. By reducing the need for multiple tracks, this feature helps users to work smarter and faster, allowing them to focus on creative storytelling and deliver engaging content.

Get started with our Video Editing for Web documentation.

Get Vibrant Colors with P3/Wide Gamut Support

Unlock a broader color range with P3 color space support on iOS and Android devices. This enhancement ensures your designs and content are more vibrant, accurate, and true to life, giving you richer color options for all your projects.

Set up color spaces with our documentation for iOS and Android.

💡 The Canvas menu appears when you select a design block, letting you edit its settings, like duplicating a selected image.

Building on mobile configuration introduced in our last release, we’ve extended customization to the editor Canvas Menu for Android. This update gives you full control over how the menu and its items appear and behave in the editor:

Customization: Configure visibility, transition effects, and decoration. Adjust the style of the divider between buttons.
Item List Control: Modify, add, reorder, or hide menu items.
Flexible Button and Divider Configuration: Adjust button appearance, functionality, and divider styles to match your design requirements.

More customizations will be available soon, including for iOS.
Explore the documentation.

Improvements

Switch the Editor Language without Restarting

Easily switch languages in our web editor without restarting it. This update lets you update the UI language instantly via API, streamlining localization. Check our documentation, and translate right away!

Upcoming

Record Reactions to Videos on Android
We’re excited to announce that Reactions will soon be available on Android. Already live in CE.SDK on iOS, this feature (popularized by platforms like TikTok and Instagram) allows users to capture and share their real-time responses to videos. Enhance engagement and provide content creators with immediate, interactive feedback, making your video content even more dynamic.

View the Changelog. Thanks for reading!

3,000+ creative professionals gain early access to new features and updates. Don’t miss out, and subscribe to our newsletter.

How To: Video Generation With Javascript

Antonello — Wed, 22 Jan 2025 09:53:00 GMT

Follow this tutorial to learn how to programmatically create videos in JavaScript directly in the browser.

In 2025 video is firmly established as powerhouse for engagement, gaining even more traction with the rise of short-form videos. After all, as humans, we naturally resonate with video content more than any other type of media.

The problem is that video generation is challenging, and doing it within the browser through JavaScript makes things even more complex. Fortunately, production-grade solutions like CreativeEditor SDK (CE.SDK) make the process not only possible but also easy to implement.

In this guide, you will learn how to programmatically generate videos in the browser via JavaScript using CreativeEditor SDK. We will cover various applications, including merging videos, adding audio tracks, and even integrating AI features.

Let’s dive in!

Why Programmatic Video Generation Matters

Videos are dominating social media, marketing, and online platforms such as e-commerce platforms (e.g., to showcase products). Numerous studies have proven that video is the most effective form of media for marketing, as it resonates deeply with us.

As the demand for personalized and dynamic content increases, traditional video production methods are becoming inefficient. The proliferation of channels with different demands on size and video quality as well as the need to create personalized videos at scale means that automated video creation is becoming indispensable. Most existing solutions run batch processing of videos on the server after gathering input data from various source. However, while server-side processing quickly becomes costly and introduces significant communication overhead client devices have become more powerful than ever.

CreativeEditor SDK bridges that gap by simplifying programmatic video generation in JavaScript within modern browsers. With CE.SDK, you can automate repetitive tasks and define scalable, personalized, and engaging video creation workflows, ensuring each generated video resonates with its target audience.

Get Started with CE.SDK for JavaScript

Now that we established why programmatic video generation is important, you are ready to get started with the tool designed for it: CreativeEditor SDK.

Learn how to integrate CE.SDK for video editing into a vanilla JavaScript application!

You can access the code for this tutorial on Github.

Requirements

The only prerequisites to follow this tutorial are:

A modern browser: CreativeEditor SDK runs directly in your browser’s JavaScript engine, so no additional setup is required. Any browser supporting WASM is enough.
A CreativeEditor SDK license: If you do not have one yet, sign up for a free trial.

If your web application uses Node.js, ensure you have the latest stable versions of both Node.js and NPM installed.

Import the library

In a vanilla JavaScript application, create a JavaScript module file (e.g., video-editor.js) and import the CreativeEditor SDK engine library inside it:

import CreativeEngine from '<https://cdn.img.ly/packages/imgly/cesdk-engine/1.42.0/index.js>';

Note: In this example, the SDK is served from our CDN for convenience. In a production environment, it is recommended to host all assets and libraries on your own servers for improved control and performance.

To use the video-editor.js module in an HTML page, import it with this line:

<script type="module" src="video-editor.js"></script>

Note the type=“module” attribute tells the browser to treat the script as an ES6 module, so that you can use import and export statements.

Alternatively, if you are using a module bundler like Webpack, Rollup, Parcel, or Vite, add the @cesdk/engine npm package to your project’s dependencies:

npm install @cesdk/cesdk-js

In this case, you can then import the headless SDK into your JavaScript code as follows:

import CreativeEngine from '@cesdk/engine';

Well done!

Setting Up Your Environment

In your JavaScript module, right below the library import, initialize the CreativeEditor SDK headless engine using the following code:

// src/video-editor.js

import CreativeEngine from '<https://cdn.img.ly/packages/imgly/cesdk-engine/1.42.0/index.js>';

// your CE.SDK license and user configs
const config = {
  license: '<YOUR_LICENSE>',
};

CreativeEngine.init(config).then(async (engine) => {
  // attach the engine canvas to the DOM
  document.getElementById('cesdk_container').append(engine.element);

  // do something with your instance of CreativeEditor SDK...

  // detach the engine and clean up its resource
  engine.element.remove();
  engine.dispose();
});

Replace <YOUR_LICENSE> with the license key provided in your CreativeEditor SDK subscription.

Ensure that the HTML page importing video-editor.js contains the following <div> element with the cesdk_container ID:

<div id="cesdk_container" style="width: 100%; height: 100vh;"></div>

The CreativeEngine.init() method will mount the editor engine component within this div.

Congratulations! You have successfully integrated the video editing engine.

Load a Video

Use the logic below inside the body of the CreativeEndine.init() callback function to load a video in CE.SDK:

// initialize a new video scene
const scene = engine.scene.createVideo();

// create a page block and attach it to the scene
const page = engine.block.create('page');
engine.block.appendChild(scene, page);

// set the dimensions of the video page
engine.block.setWidth(page, 720);
engine.block.setHeight(page, 1280);

// create a graphic block to hold the video content
// and set its shape to a rectangle
const video = engine.block.create('graphic');
engine.block.setShape(video, engine.block.createShape('rect'));

// create a fill type for the video and set the video file URI
const videoFill = engine.block.createFill('video');
engine.block.setString(
  videoFill,
  'fill/video/fileURI',
  '<https://cdn.img.ly/assets/demo/v2/ly.img.video/videos/pexels-drone-footage-of-a-surfer-barrelling-a-wave-12715991.mp4>'
);

// apply the video fill to the video graphic
engine.block.setFill(video, videoFill);

// create a track block to manage the video timeline
// and attache the track to the page and the video graphic
// to the track
const track = engine.block.create('track');
engine.block.appendChild(page, track);
engine.block.appendChild(track, video);

// make the track cover the parent block
engine.block.fillParent(track);

This initializes a video scene using createVideo() and creates a "graphic" block to display the video in a rectangle, adding it to the page. Next, it loads a video from a remote resource and appends it as a track to the parent block.

For a complete explanation of how the above code works, refer to the documentation.

If you comment out the cleanup part, this is what you should see in your browser application:

Notice how the video from the remote URI has been loaded into the

element where CreativeEditor SDK is attached to the page’s DOM.Time to explore more programmatic video generation features!

Programmatic Video Generation Features

Follow the use cases below to see how CE.SDK makes automated video generation in JavaScript easier.Generate Video in Different Formats CE.SDK supports video export in MP4 format (with MP3 audio) with different resolutions and aspect ratios.To export a video with a specific aspect ratio, first adjust the width and height of the parent block to which the video block is attached:

// specify an aspect ratio of 9:16 (ideal for vertical videos)
engine.block.setWidth(page, 720);
engine.block.setHeight(page, 1280);

In this case, the video will be automatically adjusted to fit the specified dimensions. Horizontal videos will be cropped to match the defined vertical outline of the block, while vertical videos will be rendered as they are within the specified block.

Next, you can export the video with the configured 9:16 aspect ratio and a given resolution with:

// define the MIME type for the video export (MP4 format)
const mimeType = 'video/mp4';

// define a callback function to track the progress of video rendering and encoding
const progressCallback = (renderedFrames, encodedFrames, totalFrames) => {
  console.log(
    'Rendered', // log the number of rendered frames
    renderedFrames,
    'frames and encoded', // log the number of encoded frames
    encodedFrames,
    'frames out of', // log the total frames to be processed
    totalFrames
  );
};

// define the video export options
const videoOptions = {
  duration: 5, // video duration in seconds
  framerate: 30, // video framerate (frames per second)
  targetWidth: 480, // target video width in pixels (for the resolution)
  targetHeight: 852, // target video height in pixels (for the resolution)
};

// export the page as an MP4 video
const blob = await engine.block.exportVideo(
  page, // the page containing the design video block
  mimeType, // the MIME type for the video (MP4)
  progressCallback, // the callback to track video rendering progress
  videoOptions // the options for the video export
);

// create an anchor element to trigger the video download
const anchor = document.createElement('a');
anchor.href = URL.createObjectURL(blob);
anchor.download = 'video.mp4'; // output video name
anchor.click();

The exportVideo() function produces a blob video file of the specified MIME type. Note that the export occurs over multiple iterations of the update loop, with a frame being encoded in each iteration.

The targetWidth and targetHeight options are optional. If used, the video will be resized to fit the target dimensions while maintaining its aspect ratio. This is useful for setting a specific resolution for the output video while preserving the parent block’s width and height on the output video. If omitted, the produced video will match the resolution of the width and height specified with setWidth() and setHeight(), respectively.

The progressCallback() function will track and display the video generation progress in the browser’s console:

The final lines of the above code snippet trigger the download operation, mimicking the action of clicking a link to download a resource. This is a common JavaScript technique to programmatically trigger a file download.

Once the export process completes, the output video file (video.mp4) will be automatically downloaded to the browser’s download directory. Open its properties and this is what you will see:

Note the video file’s resolution and the fact that its duration is set to 5 seconds, as defined by the duration attribute.

These options enable you to easily create clips with different lengths, aspect ratios, and resolutions, optimized for various audiences and social media platforms like LinkedIn, Instagram, Facebook, X, TikTok, YouTube, and more.

Add Audio Tracks

Adding an audio track for narration or background music to your clip is straightforward. First, create an “audio” block and add your audio resource to it:

const audio = engine.block.create('audio');
engine.block.appendChild(page, audio);
engine.block.setString(
  audio,
  'audio/fileURI',
  'https://cdn.img.ly/assets/demo/v1/ly.img.audio/audios/far_from_home.m4a'
);

Next, you can easily adjust the volume, and apply fade-in and fade-out effects as follows:

// set the volume level to 80%
engine.block.setVolume(audio, 0.8);

// start the audio after 1 second of playback
engine.block.setTimeOffset(audio, 1);

// set the audio block's duration to 5 seconds
engine.block.setDuration(audio, 5);

Here is breakdown of the functions used above:

setVolume(): Adjusts the audio volume of the block, with a range from 0 (0%) to 1 (100%).
setTimeOffset(): Sets the time offset of the block relative to its parent. This determines when the block starts playing in the timeline.
setDuration(): Sets the playback duration of the audio block in seconds, defining how long the block will play during the scene.

In this example, the audio starts at the 1-second mark and plays for 5 seconds (from 1s to 6s), with the volume set to 80% of the original.

Merge Videos

To stitch multiple video clips together in CE.SDK, you first need to import and create separate video blocks:

// create the first video block
const video1 = engine.block.create('graphic');
engine.block.setShape(video1, engine.block.createShape('rect'));
const videoFill1 = engine.block.createFill('video');
engine.block.setString(
  videoFill1,
  'fill/video/fileURI',
  'https://cdn.img.ly/assets/demo/v2/ly.img.video/videos/pexels-drone-footage-of-a-surfer-barrelling-a-wave-12715991.mp4'
);
engine.block.setFill(video1, videoFill1);

// create the second video block
const video2 = engine.block.create('graphic');
engine.block.setShape(video2, engine.block.createShape('rect'));
const videoFill2 = engine.block.createFill('video');
engine.block.setString(
  videoFill2,
  'fill/video/fileURI',
  'https://cdn.img.ly/assets/demo/v2/ly.img.video/videos/pexels-kampus-production-8154913.mp4'
);
engine.block.setFill(video2, videoFill2);

At this point, you have two different video clips loaded from separate video files.

Now, assume you want to produce a 10-second video that includes both clips. Start by setting the duration of the page block to 10 seconds with the setDuration() method:

engine.block.setDuration(page, 10);

Then, suppose you want the first clip (video1) to play for 7 seconds, and the second clip (video2) to fade in after that and play for 3 seconds (from second 1 to second 4 of the original second clip). This is how you can set that up:

// set the duration of the first and second video clips
engine.block.setDuration(video1, 7); // video1 plays for 7 seconds
engine.block.setDuration(video2, 3); // video2 plays for 3 seconds

// wait for the second video to be fully loaded before applying the trim options
await engine.block.forceLoadAVResource(videoFill2);
// trim the second video to start at 1 second and play up to second 4
engine.block.setTrimOffset(videoFill2, 1);
engine.block.setTrimLength(videoFill2, 4);

// add a fade-in animation to the second video
const fadeInAnimation = engine.block.createAnimation("fade");
engine.block.setInAnimation(video2, fadeInAnimation);

These are the methods used above:

setDuration(): Defines the duration (in seconds) that a video block will be active during playback.
**forceLoadAVResource()**: Ensures that the audio or video resource (in this case, video2) is fully loaded before applying trim operations.
setTrimOffset(): Sets the start point (in seconds) within the video where playback begins.
setTrimLength(): Defines the length of the video to be played from the trim offset.
createAnimation(): Creates a fade-in animation for the second video (video2) to smoothly transition into the clip.

Finally, add both video blocks to the track block for playback:

const track = engine.block.create("track");
engine.block.appendChild(page, track);
engine.block.appendChild(track, video1); // add the first video to the track
engine.block.appendChild(track, video2); // add the second video to the track
engine.block.fillParent(track);

The result will be:

Note how the second clip fades in after 7 seconds of the first clip, lasting for 3 seconds as specified.

Incorporate Text Variables

Now, suppose you want to display cool words or messages in your videos. You can easily achieve this by adding a "text" block to your page, as shown below:

const text = engine.block.create("text");
// set the vluae of the text variable
engine.block.replaceText(text, "Surfing\nis\nCOOL!");
// set the text color to white and semi-transparent
engine.block.setTextColor(text, { r: 255.0, g: 255.0, b: 255.0, a: 0.8 });
// set the font size for the text
engine.block.setTextFontSize(text, 30);

// positioning the text on an absolute position on the video
engine.block.setWidthMode(text, "Auto");
engine.block.setHeightMode(text, "Auto");
engine.block.setPositionX(text, 130);
engine.block.setPositionY(text, 800);

// append the text block to the page
engine.block.appendChild(page, text);

The video canvas will now display a white “Surfing is COOL!” message as follows:

In this example, the content of the text block is static. However, it can easily be retrieved from a database or any other source for programmatic video generation with different messages, including personalized ones for tailored outreach (e.g., for custom greetings or offers).

Advanced Use Case: Prompt-Based Video Generation Using AI

As highlighted in our recent LinkedIn post, IMG.LY has just experimented with an MVP that turns keywords into fully edited short videos, complete with a script, voiceover, and visuals, all automated.

That is made possible by integrating CE.SDK with an LLM for coding and scriptwriting, ElevenLabs for voice synthesis, and Flux (via fal.ai) for visuals. Specifically, the LLM uses CE.SDK for video composition, animation, and rendering.

This is just the beginning, showing how AI can be combined with CE.SDK to programmatically generate video scripts, or even templates. A possible scenario would be to use the Creative Editor SDK plugin API to extend CE.SDK with custom plugins that adds AI directly to the design editor engine.

For example, you could develop a plugin with an LLM integration that allows users to input a prompt like: “Create a 15-second video with the text ‘Welcome to our App!’ and upbeat background music.” The LLM will process the request, using Creative Editor SDK to automate video creation by leveraging its powerful features.

Alternatively, the AI could produce a video that you can then automatically edit and enhance using CE.SDK capabilities. One thing is certain: the possibilities are endless with CE.SDK + AI!

Conclusion: The Future of Programmatic Video Generation

CE.SDK for JavaScript offers exceptional versatility in automating video workflows, allowing you to easily generate, edit, and render videos with just a few lines of code. You can stitch multiple video files, add background audio tracks, and incorporate dynamic text variables to finally produce videos in the browser, all thanks to the CE.SDK headless engine.

Looking ahead, AI integration with CE.SDK is unlocking even more powerful possibilities, such as prompt-based video content creation and automatic editing directly in the browser.

With the headless engine provided by CreativeEditor SDK, programmatic video generation in JavaScript can be implemented in just a few minutes. Explore the video capabilities of CE.SDK and dive into the docs.

Stay tuned for more updates, and please reach out if you have any questions. Thank you for reading.

Over 3,000 creative professionals gain early access to our new features, demos, and updates. Don’t miss out, and subscribe to our newsletter.

A Modern JavaScript Video Editor: Integration Guide

Antonello — Thu, 16 Jan 2025 15:29:04 GMT

Discover how to integrate IMG.LY’s JavaScript video editor into your web application and unlock its full range of features.

Video content has always been a powerhouse for engagement, becoming even more impactful with the rise of short-form videos. Platforms like YouTube and TikTok have built their empires on video content, as that remains the preferred medium for both creating and consuming information.

Not only have those platforms changed how we consume information, but they have also transformed user behavior. We do not just want to watch videos. We want to create them, expecting the process to be both simple and professional.

Here is why embedding video creation and editing capabilities directly into your website, web or mobile application can significantly boost user engagement and satisfaction.

In this guide, you will learn how to integrate a JavaScript video editor using CreativeEditor SDK (CE.SDK). We will explore when it is most useful and what customizations it supports to suit your unique needs.

Want to jump into the code? For a quick start, visit our GitHub repository.

Let’s dive in!

Why Does Your JavaScript Web App Need a Video Editor?

The world of content is constantly evolving, from text to images, from static visuals to long-form videos, and now from long videos to bite-sized clips. What stays consistent is that video content resonates deeply with us as humans.

No wonder, TikTok tops global app download charts, while YouTube, Facebook, and Instagram rank among the most visited sites right after Google. These platforms have not just built their empires on video. They have revolutionized how we think about video production.

They set the gold standard by enabling anyone to create professional-looking videos with minimal effort, offering tools for adding filters, overlays, audio, and text in just a few clicks. This ease of use has made intuitive video editing a must-have for web applications featuring social media integration, marketing ambitions, or content management capabilities.

By embedding a user-friendly JavaScript video editor in your application, you can ride this trend, providing a feature billions of users are already familiar with. The payoff? Lower barriers to user-generated content, enhanced engagement, and wider content distribution.

CreativeEditor SDK from IMG.LY delivers a seamless, high-performance, and feature-rich JavaScript solution for in-browser design, video, and photo editing, ready to meet your users’ expectations.

Getting Started: Adding a Video Editor in JavaScript

Follow this step-by-step guide to learn how to integrate a JavaScript video editor into your site using CreativeEditor SDK.

Get started with integrating CE.SDK in a vanilla JavaScript application!

Requirements

CreativeEditor SDK runs directly in your browser’s JavaScript engine, so no specific prerequisites are needed.

If your web application relies on Node.js, ensure that you have the latest stable versions of Node.js and NPM installed.

Installation and Library Import

In a vanilla JavaScript application, create a JavaScript module file (e.g., video-editor.js) and import the CreativeEditorSDK library:

import CreativeEditorSDK from '<https://cdn.img.ly/packages/imgly/cesdk-js/1.42.0/index.js>';

Note: In this example, the SDK is served from our CDN for convenience. In a production environment, you should host assets on your own servers for improved control and performance.

Alternatively, if you are using a bundler like Webpack, Rollup, Parcel, or Vite, and you want to integrate the CreativeEditor module within your existing project, simply add the @cesdk/cesdk-js npm package as a dependency to your project:

npm install [@cesdk/cesdk-js](<https://www.npmjs.com/package/@cesdk/cesdk-js>)

You can then import it as below:

import CreativeEditorSDK from "@cesdk/cesdk-js";

Well done!

Initialize the Video Editor

In your JavaScript module, right below the library import, initialize CreativeEditor SDK with the following logic:

// src/video-editor.js

import CreativeEditorSDK from '<https://cdn.img.ly/packages/imgly/cesdk-js/1.42.0/index.js>';

// your CE.SDK license and user configs
const config = {
  license: '<YOUR_LICENSE>',
  userId: '<YOUR_USER_ID>',
};

CreativeEditorSDK.create('#cesdk_container', config).then(async (editor) => {
  // do something with your instance of CreativeEditor SDK...

  // when done, clean up and free up resources
  editor.dispose();
});

Replace <YOUR_LICENSE> with the license key you get from CreativeEditor SDK and <YOUR_USER_ID> with your user ID. The user ID is optional and helps track MAUs (Monthly Active Users) across different devices.

To integrate the video-editor.js file, import it into your HTML page like this:

<script type="module" src="video-editor.js"></script>

Note the type=“module” attribute. This tells the browser to treat the script as an ES6 module, so that you can use import and export statements within your JavaScript code.

Make sure the same HTML page includes the following div element where the video editor will be embedded:

<div id="cesdk_container" style="width: 100%; height: 100vh;"></div>

This will load the video editor with the default video preset, giving users the ability to trim, cut, apply filters, add text overlays, include music, and more to enhance their videos.

Congratulations, you have successfully integrated the video editor!

CreativeEditor SDK: JavaScript Video Editing API for Web, Desktop, and Mobile

You just saw how to integrate the CE.SDK into a browser application. Yet, do not forget that the SDK supports a wide variety of frameworks and libraries, including mobile and web applications.

Web Framework Support

CE.SDK is designed to work with several popular JavaScript frameworks and libraries. For more detailed integration instructions, take a look at the official page on the documentation for each framework:

Additionally, the SDK is available for headless mode and works well in a Node.js environment.

Cross-Platform Support

One of the standout features of CreativeEditor SDK is its cross-platform philosophy. This extends beyond JavaScript ecosystems for the web, and includes mobile (iOS and Android) and desktop, thanks to support for:

That is made possible through a powerful engine that powers your app via features like:

Unified API: The same underlying API is shared across web, desktop, iOS, and Android applications.
Cross-Platform Feature Parity: CE.SDK guarantees that features are available consistently across platforms. Since core functionality is implemented at the engine level, features you use on one platform will be available on others, with only slight differences in release timelines.
Interoperable Designs: Designs created in the editor component on one platform work on all platforms supported by CE.SDK. Whether a user is designing on iOS, Android, or the web, their design files can be imported seamlessly between platforms.

Native iOS and Android libraries, along with other frameworks, are also available. Learn more in our documentation!

JavaScript Video Editor Use Cases

It is time to explore some key use cases. Let’s see how the unique features of the CreativeEditor SDK can cover many scenarios!

CreativeEditor SDK features what users need to create engaging and easy-to-share videos optimized for popular social media platforms like Instagram, TikTok, YouTube, LinkedIn, and more. This is possible due to a large set of capabilities, which include:

Reusable Templates: Users can either design their own templates and share them with the community or download those created by others. This variety of ready-to-use templates for different scenarios helps users skip the daunting blank canvas to bring their ideas to life. Explore various templates in our video editor demo.
Music and Audio Library: In CE.SDK, adding audio to videos is simple and intuitive. Given the impact of audio, users can set the tone with music or narration in just a few clicks, creating a more immersive atmosphere for their content.
Animations: The SDK includes a wide selection of animations to enhance the design elements positioned within video scenes.

After creating a video within your web app, users can export it directly from CE.SDK in the appropriate formats and aspect ratios for their target social media platforms.

Product Showcase for E-Commerce Applications

Images have long been a simple yet powerful tool for showcasing products on e-commerce platforms, and sites like eBay, Amazon, AliExpress, and Walmart have relied on them for years (and still do!).

However, in recent years, customer expectations have shifted. Today’s buyers crave more interactive and immersive ways to visualize products. They want to see how items look and perform in real-world settings.

Short, dynamic product videos offer a superior presentation experience. According to an analysis by Firework, video content increases purchase likelihood by 51%, highlighting its undeniable impact on buying decisions. This is why many sellers have already embraced video to enhance product displays.

With a CE.SDK-powered JavaScript video editor, producing compelling videos that deliver a seamless, multi-device shopping experience becomes effortless. Plus, reusable templates help you generate unique product videos while maintaining brand consistency.

Engaging Video Messages for Sales Outreach

Marketers and sales teams know that cold emails often come across as impersonal and boring. The solution to this challenge? Personalized video messages!

Customized video messages are proving to be an effective outreach tool, with studies showing significantly higher conversion rates.

With CreativeEditor SDK, sales and marketing teams can easily create personalized videos tailored to individual clients or leads, making their outreach more engaging and impactful. Unlike traditional email campaigns, personalized videos create a sense of real, one-on-one interaction.

Additionally, the SDK’s translation features help sales teams break down language barriers, allowing them to reach a global audience. This ensures that the personalized messages resonate with clients from different regions.

Explore this feature in our translation and internationalization demo.

Digital Asset Management

DAM, short for Digital Asset Management, is the process of organizing, storing, and managing digital assets, including video content. As your business grows, an effective DAM system becomes key to enabling multiple teams to easily access and collaborate on assets across different projects.

CreativeEditor SDK backs DAM workflows by offering powerful tools to organize, edit, and repurpose video content through an embeddable, centralized interface. Its core JavaScript video editor component empowers users to:

Effortlessly adapt and customize video assets.
Ensure brand consistency while automating updates across various assets.
Edit content efficiently within a collaborative environment.

Although CE.SDK does not provide built-in role-based permissions, it integrates smoothly with backend systems to control user access. In particular, its flexible architecture supports the definition of custom hooks for permission checks to customize the editor’s behavior.

Automated Video Creation

CreativeEditor SDK streamlines the automation of asset creation by enabling users to generate on-brand video content variations for multiple channels. Thanks to adaptable templates, dynamic text variables, and lockable designs, the SDK ensures consistency and brand compliance over all produced videos.

Given a template, the creative engine automates the generation of countless personalized assets by populating them with resources from various data sources. That eliminates manual effort, reduces design bottlenecks, and accelerates production, making it perfect for scaling campaign-specific videos or building dedicated marketing tech.

Customizing your CreativeEditor SDK Instance

CreativeEditor SDK is not a one-size-fits-all JavaScript video editor. Instead, it is built for flexibility, offering a wide set of customization options to tailor the editor to your unique requirements and brand identity. These include:

Theming Options: Match the editor’s look to your app’s style with built-in themes, generate new ones using the theme generator, or create custom themes manually. Explore theming possibilities in our dedicated demo.
Custom UI: Build entirely bespoke JavaScript video editing user interfaces that align with your specific use case.
Media Library Integration: Set up sortable, flexible media libraries and integrate popular third-party libraries like Unsplash and others. Get easy access to external assets.
Localization: Adapt the editor to different languages and regions with full i18n support, letting you extend or overwrite any language setting.
Toolbar Customization: Rearrange toolbar elements, update icons, or rename tools to deliver a tailored user experience.

With these customization options, CreativeEditor SDK ensures your video editor is as unique as your JavaScript application.

Conclusion

Implementing a JavaScript video editor in your web app can greatly improve the user experience and provide a powerful feature to drive higher engagement, retention, and even product distribution. This is true whether you are developing an e-commerce platform, a social media app, a tool for influencers, or a SaaS web solution.

With the CreativeEditor SDK, adding a fully customizable video editing experience to your web application takes only a few minutes. Explore CE.SDK’s video capabilities and dive into the docs.

Stay tuned for more updates, and please reach out if you have any questions. Thank you for reading.

Over 3,000 creative professionals gain early access to our new features, demos, and updates. Don’t miss out, and subscribe to our newsletter.

A Modern React Design Editor: Integration Guide

Antonello — Thu, 16 Jan 2025 14:38:04 GMT

Learn how to integrate IMG.LY’s design editor for React into your web app and explore various use cases and customization options.

It is no surprise that content asset generation is a key for businesses in marketing and promotion, as well as for users on social media. Whether it is images or videos, these are the types of posts we engage with and relate to the most as human beings, much more than plain text.

Most content that resonates with an audience is not created from scratch. Instead, it often comes from pre-designed designs that are reused, repurposed, and adapted to fit different content or goals. That is why adding a design editor to your React application is so valuable.

In this guide, you will learn how to build a React design editor using the CreativeEditor SDK (also known as CE.SDK). We will also cover when this is useful and the customization options the editor supports.

For a quick start, check out the GitHub repository with the full code integration.

Let’s dive in!

Why Add a Design Editor to Your React App?

Before answering the question, you must understand what a design is and how helpful it can be. In this context, a design can be defined as a set of predefined assets, text overlays, placeholders, and other elements arranged in a specific layout.

Users can use designs to create images or videos with a particular structure and for specific use cases at scale. Thus, a design editor would enable your users to create personalized assets, and in the case of a React application, directly within the browser.

Building such a platform is not a piece of cake. A robust design editor requires numerous features, such as intuitive drag-and-drop functionality, template management, image manipulation tools, and more. It also generally requires integrations with third-party services and dependencies to enhance its capabilities.

Fortunately, an all-in-one solution like CreativeEditor SDK from IMG.LY makes it easy to embed this powerful design editor into your application. It supports various programming languages and technologies, including React. With extensive features and customization options, CE.SDK has no limits on the types of designs users can generate.

CE.SDK features adaptable design tools for diverse industries and use cases, such as social media post creation, digital asset management, and AI-assisted content personalization. That is why it is used by hundreds of customers, including governments, Fortune 500 companies, and dozens of startups.

CreativeEditor SDK is trusted by companies like Swiss Post, Brother, Optimizely, HP, and many more.

How to Integrate a Design Editor in React

Follow this step-by-step tutorial to see how to quickly set up a React design editor in your web app using CreativeEditor SDK. For a complete guide, read our tutorial on building a Canva alternative in React using CE.SDK.

Integrate CE.SDK into your React app using the instructions below!

Requirements

Before getting started, ensure that you meet the following prerequisites:

The latest LTS version of Node.js: If you do not have it installed, download it from the official site and follow the installation instructions.
A React 18+ application: If you do not have one set up, follow the instructions below to create it.
A CreativeEditor SDK license: In case you do not have a license, sign up for a free trial or buy a license.

If you do not have a React project set up yet, create a new one using Vite:

npm create vite@latest my-react-video-editor-app -- --template react

Open your project in a JavaScript IDE, and you are all set to start coding!

Install CE.SDK

Install CE.SDK by adding the @cesdk/cesdk-js package to your project’s dependencies:

npm install @cesdk/cesdk-js

Your package.json file will now list it as a dependency:


"dependencies": {
  "@cesdk/cesdk-js": "^1.41.1",
  "react": "^18.3.1",
  "react-dom": "^18.3.1"
}

Create the Design Editor Component

In the src folder of your React project, create a DesignEditor.jsx file. This will house your CE.SDK-based React design editor component.

Inside it, import CreativeEditorSDK from @cesdk/cesdk-js:

import CreativeEditorSDK from '@cesdk/cesdk-js';

Next, initialize your video editor React component with the following logic:

// src/DesignEditor.jsx

import CreativeEditorSDK from '@cesdk/cesdk-js';
import { useEffect, useRef, useState } from 'react';

// your CE.SDK license and user configs
const config = {
  license: '<YOUR_LICENSE>',
  userId: '<YOUR_USER_ID>',
};

export default function DesignEditor() {
  const cesdk_container = useRef(null);
  const [cesdk, setCesdk] = useState(null);

  useEffect(() => {
    // if the CE.SDK container has already been initialized
    if (!cesdk_container.current) {
      return;
    }
    let cleanedUp = false;

    // initialize a CE.SDK instance
    let instance;
    CreativeEditorSDK.create(cesdk_container.current, config).then(
      async (_instance) => {
        instance = _instance;
        if (cleanedUp) {
          instance.dispose();
          return;
        }

        // customize the design editor behavior...

        setCesdk(instance);
      }
    );

    // clean up the CE.SDK instance
    const cleanup = () => {
      cleanedUp = true;
      instance?.dispose();
      setCesdk(null);
    };

    return cleanup;
  }, [cesdk_container]);

  return (
    <div
      ref={cesdk_container}
      style={{ width: '100vw', height: '100vh' }}
    ></div>
  );
}

Replace <YOUR_LICENSE> with the license key from CreativeEditor SDK, and <YOUR_USER_ID> with your user ID to optionally track monthly active users (MAUs) across different devices.

The component defined above embeds CreativeEditor SDK into an HTML <div> element. See how to customize and use it in the docs.

Use the Component

You can now import the DesignEditor component as below:

import DesignEditor from "./DesignEditor";

Then, add it to the DOM as follows:

<DesignEditor />

This will open the design editor with the preset, enabling users to place assets, set constraints, and perform other tasks to create their final designs.

React Design Editor Use Cases

Now that you are familiar with integrating a React design editor into your web app. It is time to explore key use cases and see how CreativeEditorSDK can cover many scenarios.

CreativeEditorSDK offers a wide suite of tools to help users easily create high-quality, personalized, on-brand social media assets in your React application.

With a vast library of professionally designed templates, it streamlines the design process and ensures consistency across multiple platforms. Users can efficiently produce engaging content tailored for

Key features include customizable dimensions, filters, placeholders, and text overlays. These give users what they need to craft content optimized for various social networks, such as Instagram, X, LinkedIn, and more. That is possible thanks to platform-specific export options to produce ready-to-upload assets.

The SDK’s capabilities extend beyond static images, supporting the creation of dynamic short videos, stories, and reels. Do not forget that CE.SDK can also function as a modern React video editor.

Digital Asset Management

DAM, short for Digital Asset Management, is the process of organizing, storing, and efficiently retrieving a wide range of digital files.

CreativeEditorSDK supports DAM by helping teams centralize asset creation and editing directly within an embeddable React design editor, eliminating the need for external tools. This is achieved through features like pre-defining and locking branding elements (including logos, fonts, and color schemes) to ensure brand consistency across all generated content.

Additionally, CE.SDK offers :

Design validation tools to prevent the creation of misaligned assets.
Automatic asset generation to streamline content variations.
Robust asset management features for handling both generated content and external assets retrieved via third-party APIs.

Marketing Automation

Businesses can use CreativeEditor SDK to scale marketing campaigns through dynamic, unique, customizable design templates. In detail, this design editor allows you to generate campaign materials (like personalized flyers, ads, or banners) quickly and at scale.

Key features for marketing automation include:

Automation: Integrate with CRM or marketing tools to auto-populate templates.
Dynamic Templates: Easily create content with placeholders or custom text variables for personalized outputs and A/B testing.
Brand Compliance: Lock assets like logos and fonts for consistent messaging and branding.

AI-Powered Applications

AI is rapidly becoming a key component of more and more applications, with many integrating it in some form. Still, a critical step is often overlooked: manual editing and supervision.

CreativeEditorSDK bridges that gap by seamlessly integrating with AI-powered workflows while guaranteeing users the ability to manually define constraints, placeholders, and rules that the AI-driven automation process must follow.

Example of how a CE.SDK-powered React design editor could work with AI includes:

Automated template suggestions based on industry or use case.
Prompt-based content generation where users describe their needs, and the app delivers a draft design.
Automated content optimization, where the AI evaluates design elements and suggests improvements for readability and visual appeal.

AI-generated content is impressive, but it is essential to find the right balance between automation and manual control in the content generation process for the best results.

Product Mockups

CreativeEditorSDK simplifies the creation of product mockups, packaging designs, and marketing materials, including printable brochures and business cards.

CE.SDK features powerful customization tools and flexible capabilities, including alignment guides and snap tools to ensure perfect positioning. Additionally, color pickers and vector shape editors enable you to explore and create professional design concepts.

E-Learning Materials

Thanks to CreativeEditorSDK, you can integrate a design editor into your React e-learning application.

The embeddable design editor supports the entire process of creating e-learning materials, including custom graphics like annotations, diagrams, and visual explanations, or videos recorded directly from the camera by the teachers/educators/coaches. The component also allows for the creation of unique and customized student certificates.

Those features empower educators and content creators in the education sector to craft visually engaging and impactful learning materials.

Customization Options With CreativeEditor SDK

CreativeEditor SDK does not provide a static React design design experience. Instead, it offers extensive customization options to align with your application’s branding and workflow. Key customization options are:

Theming: Adjust the editor’s theme to match your app’s design. Choose from built-in themes, generate new ones using the theme generator, or create a custom theme manually. Test themes in the demo page.
Custom UI Plugins: Add custom UI components to fit your design style through custom plugins.
Enabling/Disabling Features: Tailor the editor by enabling or disabling features like specific fonts, templates, or tools.
Toolbar Customization: Reposition toolbar elements, change icons, or rename tools to suit your needs, creating a unique editing experience.
Custom Templates, Assets, Themes: Add your own templates, assets, or themes for a more personalized user experience.
Media Libraries: Set up customizable, sortable media libraries. CE.SDK supports integration with third-party libraries, allowing easy access to external media assets.
Custom Integrations via Events: Hook into the editor’s events to trigger workflows, save progress, or integrate with external systems.
Internationalization: Adapt the design editor component for different languages and regions with full i18n support. Overwrite and extend text strings in any language.

Conclusion

CreativeEditor SDK’s compatibility with React’s component-based architecture and large set of features ensures simple integration into any project, making it ideal for a wide range of industries, from marketing and e-learning to e-commerce and social media.

A CE.SDK-based design editor offers UI customization and unique feature options to adapt to any use case. Additionally, its support for AI workflows ensures your application remains future-proof and ready to meet evolving user needs.

By following the steps outlined in this blog post, you can bring professional-level design editing features to your web users. Explore CE.SDK’s capabilities and dive into the docs.

Stay tuned for more updates, and please reach out if you have any questions. Thank you for reading.

Over 3,000 creative professionals gain early access to our new features, demos, and updates. Don’t miss out, and subscribe to our newsletter.

How I Built a Short Video Generator with AI & CE.SDK in One Day

Eray — Thu, 09 Jan 2025 11:27:13 GMT

Here’s the crux of product development in the age of LLMs: how much can AI truly accelerate the development process?

We have seen videos of solo developers building small apps entirely with AI with just a few prompts. But how does it scale to more complex development projects? As LLMs rapidly evolve, their scope and impact will only increase.

That’s why I regularly challenge myself to build a small project with the help of AI. I’m a prime candidate to test the AI productivity boost: a jack-of-all-trades (and a master of none) with a background in both design and engineering, yet no hands-on experience in the past five years. My latest challenge? Build a web-based short video generator within one day.

In this post, I’ll share the most intriguing takeaways from tackling this project.

Why a Short Video Generator?

Why focus on this idea? It’s simple: to ride the wave of a new trend. A format called “faceless” short videos is gaining traction among creators on platforms like YouTube and TikTok.

What’s fascinating about these videos is their automation: an LLM generates a script, which is then transformed into speech, images, and text assets using various AI services. These assets are automatically assembled into a cohesive video.

The general concept is compelling: It’s still generative content, but mixed with classic video composition techniques. This approach offers greater accuracy, consistency, and control over pure generative AI.

The potential to automate video production at this scale is exciting. Add its relatively low complexity and high production value, and it became the perfect topic for my challenge.

Enter CE.SDK

Another reason I chose this challenge was its compatibility with CE.SDK, our design and video editor library. CE.SDK offers a robust editing toolkit that integrates into any product with just a few lines of code. Its features, like headless mode, are ideal for automating workflows like video generation.

Most faceless video services use React-based video generation and achieve fair results. However, using CE.SDK instead of a react-based library could potentially boost the overall experience with three critical improvements:

Editable Outputs: This is huge. Full automation often needs human adjustments for fine-tuning. CE.SDK enables automated video generation while allowing manual refinement of the results.
Enhanced Visual Quality: CE.SDK has its own rendering pipeline, allowing for more nuanced visual effects and animations. When you’re competing against others in this space, it can make a huge difference if you’re able to produce higher fidelity in the visual output.
Visual Design Workflow: Create design components or even entire templates visually, and then use them via code. This authoring workflow can be extremely helpful in creating rich, interesting designs for the generated videos.

The Ground Rules

To keep the challenge focused, I set strict rules:

Time Limit: Spend no more than 12 hours on the challenge.
No Manual Coding: Avoid writing any code yourself. Everything should be built through conversations with AI.
Trust the AI: Do not read or analyze code generated by the AI. Rely entirely on its decisions.
Skip External Research: Do not read or explore the APIs you intend to use. Instead, provide links to the AI and let it determine how to use them.
Compare AI Performance: Alternate Claude Sonnet 3.5 and ChatGPT o1 for code generation to evaluate which performs better.

The Tools & Workflow

Code Editor: Cursor
Built on VSCode’s foundation, Cursor stood out as the only editor offering both an integrated chat interface and the ability to switch between different LLMs. However, with GitHub’s recent significant updates to Copilot, I’ll switch to VSCode with Copilot for future challenges.

UI Prototyping: Claude Artifacts
Rather than building the entire project in my code editor, I chose to prototype the UI directly through Claude’s web interface. The benefits were immense:

Instant results: To create an artifact, Claude streamlines development by automatically writing and compiling code while leveraging essential UI libraries and components. This automation eliminates setup time and technical overhead, allowing me to focus purely on design iterations.
Instants Variations: Claude enables rapid prototyping through parallel conversations. When a design direction didn’t quite work, I could simply start a fresh conversation with modified requirements and evaluate a new prototype. This approach helped me develop three viable concepts quickly - a pace that would have been impossible in a traditional code editor.
Quality of execution: Claude transforms rough concepts into polished, intuitive interfaces. Its suggestions often surpassed my initial ideas, offering sophisticated solutions I hadn’t considered.
Keep it clean: By prototyping outside the code editor, I kept the main project’s codebase clean and focused. This separation prevented the accumulation of experimental code and maintained the clarity of our primary development environment.

Quickly prototype your interface with Claude Artifacts.

APIs
Key APIs used in the project included:

Script Generation: Claude Sonnet 3.5 vs various ChatGPT models.
Image/Video Assets: Fal.ai Flux models.
Speech Synthesis: ElevenLabs.

Building the App: Divide and Conquer

After having prototyped the UI, I started to chat with the LLM inside the code editor so that it can code the app. To work with the AI efficiently, I followed a divide-and-conquer approach. Rather than simply asking it to “build me a video app,” I broke down the problem into manageable steps:

Generate a video script
Create an AI prompt that includes user input and examples of the desired output format. Pass this prompt to the LLM API.
Parse the script to generate assets (speech, images, text)
Parse the LLM’s response to extract image prompts and speech paragraphs. Send these to their respective APIs.
Compose the final video
Load all the generated assets into a predefined template to generate the finished video through the CE.SDK library.

After completing these steps, I was finally able to generate my first fully automated videos! With a few more tweaks and additions, I had an MVP ready within twelve hours.

The final result: A Short Video Generator

There are still some missing features, partly because I spent a significant amount of time refining the prompt to generate the video script. I also had to bend the rules occasionally: sometimes the LLM would hit a wall, and I had to read or write small snippets of code.

Key Takeaways

Engineering Knowledge Is Essential
You should have some engineering background to achieve the AI productivity boost in development.

AI doesn’t solve everything for you. You are still the architect. You provide a lot of input and guidance. AI often needs to be pointed to the right strategy. Foundational knowledge of computer science is hugely advantageous for working with AI effectively.
As mentioned, I had to read and write a few lines of code myself. Without coding experience, I would have probably not been able to progress, as the LLM was not able to.
The getting started experience is nowhere close to novice-friendly. How do you get started with a new project in a code editor that actually requires you to do the setup manually? My workaround was to create an empty project, and then ask the LLM to instruct me to use a boilerplate for react. Again, this is engineering knowledge, any novice would have hit a wall already at this point.

Claude Outperformed ChatGPT
Claude was a clear winner in the side-by-side comparison, because of three reasons:

Claude Artifacts was a game changer for UI prototyping.
It was generally better at writing and understanding code. Difficult to quantify, but in some cases Claude fixed the mess ChatGPT left in the code
Claude can process URLs, which makes working with APIs much smoother.

Who would have thought new LLMs would catch up to OpenAI so quickly after they released the first version of ChatGPT?

Complexity Slows AI
The more code in my project, the slower the overall progress. LLMs struggled with the growing complexity. Their context windows filled more quickly, and their responses became increasingly unreliable. At some point, it becomes extremely difficult to make architectural changes, especially if this affects multiple parts of the app. When trying to fix errors, you’ll often find yourself in a whack-a-mole game. While the AI would resolve one issue, it would inadvertently introduce new problems elsewhere, creating an endless loop of fixes and regressions.

Ultimately, the time invested in this challenge was well worth it. While LLMs can’t build products end to end on their own, they can significantly streamline product development when paired with the right human collaboration. The real question is whether development teams are ready to adapt their habits and explore new workflows to boost productivity.

Next Steps

This challenge has inspired me to refine and expand on this project. Future iterations will focus on harnessing CE.SDK’s unique features to push the boundaries of automated video generation.

Stay tuned for part two of this series. There’s much more to explore!

UPDATE: Read part two - a cookbook how to build your own short video creator!

Over 3,000 creative professionals gain early access to our new features, insights and updates. Don’t miss out, and subscribe to our newsletter.

CE.SDK – IMG.LY Blog

Evaluating CoDesign MCP performance in resizing

Three banners, one brief

What is gradeable and what is not

What we are doing right now

One master, five ad sizes

Which model is best for using with CoDesign MCP?

What does it mean for CoDesign MCP

CE.SDK v1.78 Release Notes

More Expressive Typography

Curve Text with a Dedicated Path Control on the Web

Improvements

Keep Edited Text Above the Mobile Keyboard

Browse Typefaces Smoothly on Slow Networks

Refresh iOS Asset Libraries Automatically

Stability and Fixes

Changelog

13 Questions Teams Ask Themselves Before Choosing IMG.LY

Why integrate an editor instead of building our own?

Can’t we just use an open-source library like Fabric.js or Konva?

How fast can we realistically be in production?

Every design edit request lands on my team. How do we scale that?

Our users aren’t designers. How do we keep results on-brand without manual review?

We already have an editor. Is replacing it worth the costs?

Will it match our product’s look and feel?

We ship on web, iOS, and Android. Do we need three different tools?

Our users want AI features. Do we have to build those too?

Where does our users’ data actually go?

If we build on this, are we locked in?

Can we depend on this provider when it matters?

How do we know it’ll work before we’re all in?

Recognize a few of these?

Starter Kits: A New Way to Use and Configure Our Editor

What changed

What are the benefits?

The full picture in one place

More control and capability over your editor

Opt in, never opt out

Two paths, your choice

The engine you rely on hasn’t changed

The Tradeoff

Getting Started

CE.SDK v1.69 Release Notes

Introducing IMG.LY Agent Skills for CE.SDK

Launch in Minutes with Production-Ready Starter Kits for Web

Let Users Bring Their PPTX & Canva Designs to CE.SDK

Professional-Grade Video Editing Across Platforms

Full Changelog

Building a CapCut-Like Video Editor with CE.SDK and AI

Build this with AI in minutes

Introduction

Architecture Overview

Step 1: Project Setup

Why so many packages?

Step 2: The React Component

Step 3: Creating the Video Scene

Browser support

Step 4: The CapCut Dark Theme

Setting the base theme

Custom CSS overrides

Responsive scale

Step 5: AI Plugin Integration

The unified AiApps approach

Provider categories explained

The proxy server requirement

Background removal

Step 6: UI Customization with the Component Order API

Adding the AI button to the dock

AI options in the canvas context menu

Custom export button in the navigation bar

Step 7: Wiring Generated Audio into the Asset Library

Step 8: Custom Labels with i18n

The Complete File Structure

What We Get Out of the Box

Going to Production

1. License key

2. Proxy server

3. Disable dry run

Key Takeaways

Resources