Performance Test Results¶
Legacy — full redo planned. All numbers below are historical. Rows labeled Kuzu are from Kuzu-backed runs; Ladybug is not re-benchmarked here. When this file is rewritten, use a chosen LLM and split graph-related overall time into (1) KG extraction (LLM) vs (2) insert into the specific graph database, so cost of the model vs cost of each backend is visible.
⚠️ Note: Benchmarks are being redone; results below are from earlier testing and will be replaced.
Test Environment: AMD 5950x 16-core CPU, 64GB RAM, 4090 Nvidia GPU, Windows 11 Pro
Infrastructure Configuration: - Vector Database: Qdrant (consistent across all tests) - Search Database: Elasticsearch (consistent across all tests) - Graph Extractor: SchemaLLMPathExtractor (extract type: schema) with schema name: default (consistent across all tests) - LLM Models: OpenAI gpt-4o-mini, Ollama llama3.2:3b - Embedding Models: OpenAI text-embedding-3-small, Ollama all-minilm - Index Management: Qdrant + Elasticsearch indexes cleared between LLM/Graph DB configuration changes, but preserved between 2-doc to 4-doc incremental tests within same configuration
Quick Summary: 6-Document Performance¶
The table below provides a quick overview of 6-document ingestion performance across different graph database and LLM combinations. See detailed breakdowns in following sections.
| Graph Database | LLM Provider | Ingestion Time | Search Time | Q&A Time |
|---|---|---|---|---|
| Neo4j | OpenAI gpt-4o-mini | 11.31s | 0.912s | 2.796s |
| Kuzu | OpenAI gpt-4o-mini | 15.72s | 1.240s | 2.187s |
| FalkorDB | OpenAI gpt-4o-mini | 21.74s | 1.199s | 2.133s |
| Neo4j | Ollama llama3.2:3b | 72.06s | ~4.2s | ~13.7s |
| Kuzu | Ollama llama3.2:3b | 43.10s | 4.945s | 10.572s |
Key Observations: - OpenAI consistently faster than Ollama across all graph databases - Neo4j + OpenAI provides fastest overall performance - Kuzu shows good performance with OpenAI - All configurations complete Q&A queries in under 15 seconds
Detailed Test Results¶
Neo4j + OpenAI¶
| Documents | Ingestion Time | Search Time | Q&A Time | Notes |
|---|---|---|---|---|
| 2 docs (cmispress.txt, space-station.txt) | 14.39s total Pipeline: 1.31s Vector: 0.81s Graph: 10.90s |
1.453s ("cmis" query) |
2.785s ("who was first with cmis") |
Initial ingestion |
| +4 docs (incremental to 6 total) (excel, pptx, docx, pdf) |
11.31s total Pipeline: 0.76s Vector: 0.26s Graph: 9.20s |
0.912s ("cmis" query) |
2.796s ("who was first with cmis") |
Incremental ingestion Graph: 49 nodes (43 Entity, 6 Chunk), 88 relationships (45 MENTIONS, 43 semantic types) |
| 6 docs (clean test) | 11.31s total Pipeline: 0.76s Vector: 0.26s Graph: 9.20s |
0.912s ("cmis" query) |
2.796s ("who was first with cmis") |
Clean 6-document test |
Neo4j + Ollama¶
| Documents | Ingestion Time | Search Time | Q&A Time | Notes |
|---|---|---|---|---|
| 2 docs (cmispress.txt, space-station.txt) | 53.66s total Pipeline: 2.08s Vector: 0.28s Graph: 50.71s |
4.252s ("cmis" query) |
13.716s ("who was first with cmis") |
Initial ingestion |
| +4 docs (incremental to 6 total) (excel, pptx, docx, pdf) |
72.06s total Pipeline: 2.14s Vector: 0.30s Graph: 69.40s |
~4.2s ("cmis" query) |
~13.7s ("who was first with cmis") |
Incremental ingestion |
Kuzu + OpenAI¶
| Documents | Ingestion Time | Search Time | Q&A Time | Notes |
|---|---|---|---|---|
| 2 docs (cmispress.txt, space-station.txt) | 14.79s total Pipeline: 1.35s Vector: 0.93s Graph: 11.16s |
1.998s ("cmis" query) |
5.005s ("who was first with cmis") |
Initial ingestion |
| +4 docs (incremental to 6 total) (excel, pptx, docx, pdf) |
15.72s total Pipeline: 0.41s Vector: 0.28s Graph: 14.40s |
1.240s ("cmis" query) |
2.187s ("who was first with cmis") |
Incremental ingestion Graph: 50 nodes (44 Entity, 6 Chunk), 96 relationships (42 LINKS, 54 MENTIONS) |
| 6 docs (clean test) | 15.72s total Pipeline: 0.41s Vector: 0.28s Graph: 14.40s |
1.240s ("cmis" query) |
2.187s ("who was first with cmis") |
Clean 6-document test |
Kuzu + Ollama¶
| Documents | Ingestion Time | Search Time | Q&A Time | Notes |
|---|---|---|---|---|
| 2 docs (cmispress.txt, space-station.txt) | 54.76s total Pipeline: 2.11s Vector: 1.00s Graph: 51.12s |
4.634s ("cmis" query) |
8.595s ("who was first with cmis") |
Initial ingestion |
| +4 docs (incremental to 6 total) (excel, pptx, docx, pdf) |
43.10s total Pipeline: 2.16s Vector: 0.26s Graph: 40.68s |
4.945s ("cmis" query) |
10.572s ("who was first with cmis") |
Incremental ingestion Graph: 39 nodes (33 Entity, 6 Chunk), 63 relationships (27 LINKS, 36 MENTIONS) |
FalkorDB + OpenAI¶
| Documents | Ingestion Time | Search Time | Q&A Time | Notes |
|---|---|---|---|---|
| 1 doc (cmispress.txt) | 17.26s total Pipeline: 3.34s Vector: 3.56s Graph: 9.13s |
1.714s ("cmis" query) |
1.625s ("who was first with cmis") |
Initial single document |
| 2 docs (cmispress.txt, space-station.txt) | 16.15s total Pipeline: 1.21s Vector: 0.88s Graph: 13.35s |
1.203s ("cmis" query) |
1.974s ("who was first with cmis") |
Two document test |
| +4 docs (incremental to 6 total) (excel, pptx, docx, pdf) |
10.74s total Pipeline: 0.51s Vector: 0.31s Graph: 8.94s |
0.933s ("cmis" query) |
1.901s ("who was first with cmis") |
Incremental ingestion |
| 6 docs (clean test) | 21.74s total Pipeline: 1.02s Vector: 3.09s Graph: 14.99s |
1.199s ("cmis" query) |
2.133s ("who was first with cmis") |
Clean 6-document test Graph: 12 nodes, 21 edges with rich entity types |
FalkorDB + Ollama¶
| Documents | Ingestion Time | Search Time | Q&A Time | Notes |
|---|---|---|---|---|
| 2 docs (cmispress.txt, space-station.txt) | 55.51s total Pipeline: 2.71s Vector: 0.27s Graph: 52.43s |
4.494s ("cmis" query) |
9.734s ("who was first with cmis") |
Initial ingestion |
| 4 docs | 52.87s total Pipeline: 2.14s Vector: 0.27s Graph: 50.21s |
4.696s ("cmis" query) |
11.747s ("who was first with cmis") |
4-document test |
(Note to avoid errors with space-station.txt and errors with multiple file processing, with falkorDB + Ollama, OLLAMA_CONTEXT_LENGTH=16384 set in system environment and TRIPLETS_PER_CHUNK=100 was set in flexible-graphrag .env)