CocoIndex Integration¶

CocoIndex is an incremental data-transformation framework with a Rust engine. Flexible GraphRAG can use it as an alternative ingest pipeline: same sources, same target databases, same UI and REST/MCP APIs — but CocoIndex owns orchestration and change processing instead of the default LlamaIndex/LangChain path.

Noticing that a source changed stays Flexible GraphRAG's job either way. The event detectors in incremental_updates/detectors/ — the filesystem watchdog, S3/GCS/Azure change feeds, Google Drive and MS Graph delta queries, Alfresco and Nuxeo audit streams — are shared: the auto-sync incremental update system drives them on the default pipeline, and here FlexibleMapView adapts the same detector to a CocoIndex LiveMapView. So switching pipelines does not change how your sources are watched, only what happens to a change once it is detected. (The exception is SOURCE_BACKEND=cocoindex, which reads through CocoIndex's own four native connectors instead — see Choosing the source.)

Turn it on in .env:

PIPELINE_BACKEND=cocoindex
ENABLE_INCREMENTAL_UPDATES=false
ENABLE_LANGFLOW_FLOWS=false

Related documentation¶

• CocoIndex Configuration — every .env setting
• CocoIndex Developer Guide — internals, APIs, customizing
• Meeting notes example — a CocoIndex example ported to this integration: custom extractor, three ways to run it
• Database Configuration — connection settings per store
• CocoIndex's own docs: overview · examples

Should you use it?¶

CocoIndex mode is optional, and the comparison is not "incremental vs. not" — the default pipeline with the incremental update system enabled already skips unchanged files via the same event detectors. The two differ in how re-work is avoided and what it costs to avoid it.

Use CocoIndex when you want:

• Step-level memoization. The default pipeline with incremental updates decides per file whether to re-ingest. CocoIndex memoizes per step: change the chunk size and only chunking
• embedding re-run — parsing is served from cache. Change the ontology and only KG extraction re-runs. This is the substantive difference.
• Cheaper reprocessing after a config change. You pay LLM/API calls only for the steps whose inputs actually changed, which matters most on large corpora.
• Automatic delete reconciliation. Remove a source file and CocoIndex's reconciler issues the deletes across every configured target, with no bookkeeping table.
• A Rust orchestration engine, with call batching for embeddings. Worth being precise about what this does and does not speed up: orchestration, change computation and batching are faster, but ingest wall-clock is usually dominated by LLM calls, embedding APIs and database writes — none of which Rust makes faster.

Stay on the default pipeline with incremental updates when:

• You need Langflow flows (not supported together — see below).
• You want Postgres-backed state. The incremental update system tracks documents in Postgres, which scales further and is easier to inspect, back up and share between processes. The open-source CocoIndex engine currently supports only local LMDB.
• You want to store less. CocoIndex's memoization is an additional store: cached step outputs live in LMDB on top of whatever is already in your vector, search, graph and RDF databases. The default pipeline keeps only per-document tracking rows.

What stays the same¶

Almost everything:

What changes¶

Who drives ingest, and where per-document state lives:

Install and enable¶

Install¶

# CocoIndex itself
uv pip install -e ".[cocoindex]"

# Optional extras
uv pip install "cocoindex[sentence_transformers]" # local GPU embeddings, no API key
uv pip install "cocoindex[litellm]"               # 100+ embedding providers — already satisfied,
                                                  # litellm is a base dependency
uv pip install "cocoindex[entity_resolution_llm]" # ENTITY_RESOLUTION=llm (see below)

python -c "import cocoindex; print(cocoindex.__version__)"   # expect >= 1.0.20

ENTITY_RESOLUTION=normalize needs no extra. llm needs one of the two extras below, and degrades to normalize with a warning without either, so it never fails an ingest — it just quietly does less.

entity_resolution_llm is the better default. It lists faiss-cpu, instructor and litellm; faiss-cpu is common to both extras and litellm is already a base dependency here, so the only difference between the two is instructor — one package for the whole API surface.

LlmPairResolver uses instructor to make the model return a validated Pydantic decision rather than prose. Our own resolver does not need it — it prompts for a bare name and maps the answer back to a candidate.

× Failed to build `valkey-glide==2.5.1`
╰─▶ Caused by: Cargo metadata failed. Do you have cargo in your PATH?

Enable¶

In .env — the last two are mutually exclusive with CocoIndex (see the warning at the top of this page), so set them explicitly if anything already turned them on:

PIPELINE_BACKEND=cocoindex
ENABLE_INCREMENTAL_UPDATES=false
ENABLE_LANGFLOW_FLOWS=false

Then:

1. Start the backend as usual (uv run start.py).
2. Ingest through the UI, REST, or MCP — how you call it does not change.
3. Or skip the server entirely and run the pipeline from the CLI — an option CocoIndex mode adds that the default pipeline does not have.

How sources work¶

Point the pipeline at a source and it keeps that source up to date. Every data source you configure — from .env, from the UI Data Source tab, or from a REST/MCP call — becomes its own coco.App with its own connection settings and its own watch root.

Two kinds of source, two change strategies:

• Detector-backed (10) — filesystem, S3, GCS, Azure Blob, Google Drive, OneDrive, SharePoint, Box, Alfresco, Nuxeo. These get a live stream: the existing FG change detector runs and feeds adds/modifies/deletes to CocoIndex as they happen. Only changed files are downloaded — listing is metadata-only.
• Snapshot-only (4) — web, Wikipedia, YouTube, CMIS. No change stream, so these are reconciled by a periodic re-scan (COCOINDEX_POLL_INTERVAL).

DATA_SOURCE picks the primary source that starts with the server:

Sources added later through the UI are stored in datasource_config and rebuilt automatically on the next restart.

Watching a folder: pass a directory path to /api/ingest with enable_sync=true and that directory becomes its own watched source, exactly as in the default pipeline. It does not have to be WATCH_DIR — that is only the default for the primary .env source and the staging area for UI file uploads.

Mixing CocoIndex and Flexible connectors¶

Inside the pipeline each stage independently uses either a native CocoIndex connector or a Flexible one (Flexible = Flexible GraphRAG's own LlamaIndex/LangChain adapters). You choose per stage; the defaults are all Flexible.

Two stages are deliberately Flexible-only. Parsing and KG extraction have no native CocoIndex equivalent that produces the multi-label, ontology-guided entity graphs this project builds, so they always run through Flexible GraphRAG even when every other stage is native.

KG extraction is also the one stage you can replace with your own code: KG_EXTRACTOR_BACKEND accepts a custom KGExtractor instead of either built-in. It still sits in the Flexible column — it is your Python running in the pipeline, not a native CocoIndex operation — and it can call ctx.builtin() to hand content it does not recognise back to the built-in extractor, so one run can cover a source that is not uniform.

If you request a native connector that isn't available for your chosen database, the pipeline logs the downgrade and falls back to the Flexible adapter rather than failing.

CocoIndex connectors this integration does not route¶

CocoIndex ships connectors beyond the ones wired up here. See its connectors reference for the current list; where they stand in this integration:

* Postgres pgvector is supported — as a native CocoIndex vector target, and through the Flexible LlamaIndex and LangChain adapters. Only the general table target is unrouted.

The stubs are deliberate: a None entry in COCO_VECTOR_REGISTRY marks a store as known-but-unimplemented, so the fallback to a Flexible adapter is explicit rather than accidental. Wiring one up is a builder function — see Extending it.

Note the asymmetry between the last two rows: Postgres, Kafka and Iggy are reachable as sources but not as targets. Those four sources have no Flexible GraphRAG equivalent, so they are capabilities this pipeline adds rather than gaps.

Running without the server¶

The pipeline is a normal CocoIndex app, so it also runs from the CLI with no FastAPI, UI, or MCP layer — useful for batch jobs and CI:

cd flexible-graphrag
cocoindex update cocoindex_integration/pipeline/app.py        # one catch-up pass
cocoindex update -L cocoindex_integration/pipeline/app.py     # stay live

It reads the same .env and runs the same parsers, chunkers, embedders, KG extractors, and store adapters. Only the HTTP layer is absent.

WATCH_DIR=../cocoindex-docs

Monitoring¶

Set COCOINDEX_ENABLE_PG_MONITORING=true to record ingest activity in the incremental Postgres database. This is bookkeeping only — it never drives change detection, and it is unrelated to the OpenTelemetry observability stack.

• cocoindex_run_log — one row per update cycle, with document counts and elapsed time.
• cocoindex_ingest_log — one row per file by default. Set COCOINDEX_MONITOR_DETAIL=stages for a row per pipeline stage when you need to see where a file stalls.