Guides · fast log search · measured July 2026 · 8 min read
How to search a huge log file fast.
You hit Enter on a search across your logs, and you wait. The tools everyone reaches for (grep and ripgrep) read the file's bytes fresh on every single query, so a pile of logs that grows into gigabytes charges you that wait again, and again, and again. This is the practical guide: how to make each search faster, and how to stop paying for the same bytes twice.
The punchline first, so you know where this lands: one content query that took ripgrep (the fastest grep there is) 93.8 seconds returned from a prebuilt index in 16 milliseconds. Same query, same 4.47-million-file machine. A minute and a half of a blinking cursor against one frame of video.
The short answer
Read fewer bytes. Or read them once.
The fastest way to search a huge log file is to read fewer bytes: for a one-off, narrow the search to the right file and match a plain string instead of a regular expression; for logs you search over and over, keep a prebuilt index that reads them once, so every later query is a look-up instead of another full scan. Everything below is a way to do one of those two things.
The distinction matters because logs are the worst case for a brute scan. They only grow (every service, every request, every day, rotated into more and bigger files), and the same incident has you searching the same window a dozen times. A scan pays the full price of the pile on each of those searches. An index pays it once.
The tools you already have. grep and ripgrep are the right reach for a one-off search. ripgrep especially, several times faster than grep because it uses every core and skips binaries. Both read the file's bytes fresh on every single run.
The trap with logs. ripgrep skips .gitignore'd and hidden files by default, and logs usually live in ignored folders, so rg "error" can silently search nothing. Pass --no-ignore (or -uu).
Why it crawls. A brute scan's cost is the number of bytes it reads. Logs only pile up, so every search re-reads a bigger haystack: the wait grows with the archive, not with your question.
The fix for repeat searches. An index reads the pile once, stays fresh as new lines append, and answers every later query as a look-up. One content query that took ripgrep 93.8 seconds returned from the index in 16 milliseconds, 58,625× on the same 4.47M-file machine.
Keep grep. Add an index. For one file or a live tail, a walker is perfect. For the same logs searched again and again (and for an AI agent grepping them to debug), the index answers first.
Why brute scan crawls, drawn
A scan's cost is the pile. A look-up's isn't.
Here is the whole idea in one picture. The horizontal axis is your log pile growing from one file to a whole archive; the vertical axis is how long one search takes. A brute scan climbs with the pile, because it has to read every byte to know what is in them. An indexed look-up stays flat on the floor: the reading already happened, so the size of the pile stops mattering.
The 93.8 s → 16 ms marker is measured. The two lines show the shape, not a timing at every pile size: the rising line is the read-everything cost model both grep and ripgrep obey; the flat line is the index. Ryzen 9 9950X3D · 64 GB DDR5 · NVMe · Windows 11 · 4.47M files.
Do this today · with the tools you have
Seven ways to make a scan hurt less.
Before we get to the index, here is the honest, no-install guide: every one of these works right now with grep or ripgrep, and each is a way to read fewer bytes. If your logs are a one-time archive you will search once, this is all you need.
Reach for ripgrep over grep. ripgrep (rg) fans the search across every core and skips binary files automatically. On a large tree of logs it is several times faster than grep -r. Author Andrew Gallant's published benchmarks show the margin plainly. New to it? Start with what ripgrep is.
Mind the .gitignore trap. Logs usually sit in ignored directories, and ripgrep respects .gitignore and hidden files by default, so it will quietly skip them. Add --no-ignore or -uu to search them anyway. The full story is in how to search the files your tools skip.
Use fixed-string mode for literals. When you are matching a plain string, not a pattern, rg -F "connection refused" (or grep -F) turns off the regex machinery and scans the literal directly.
Narrow before you scan. Search today's file, not the whole rotated archive. Point the tool at the one file or time window you need, and glob to log files with rg --glob "*.log" so you never touch the rest of the tree.
Stop at the first hit. If you only need to confirm a match exists, rg -m1 (or piping to head) returns the moment it finds one instead of reading to the end of a multi-gigabyte file.
Search compressed logs in place. Rotated logs are often .gz or .zst. rg -z (or zgrep) reads them directly, so you skip decompressing gigabytes to disk just to grep them.
Show context, not a second run. rg -C3 prints three lines around each hit. Reading the surrounding lines once beats re-running a wider search and re-reading the file again.
Every trick here makes the scan smaller or faster, and every one still re-reads the bytes the next time you ask. The moment you search the same logs twice, you have paid for them twice. That is the ceiling no flag lifts, and it is where an index comes in.
The ceiling
ripgrep is brilliant. It is also still a scan.
None of this is a knock on ripgrep. It is a genuinely great piece of engineering: a parallel walk across every core, a finite-automata regex that can't blow up into catastrophic backtracking, SIMD-accelerated scanning of literal strings. It earned its reputation, and it is the right tool for a huge share of what you do. But every one of those wins makes the read faster. To answer “which lines contain this,” ripgrep still has to open the file, pull its bytes through the CPU, and scan them. On a pile of logs, the bill scales with the size of the pile, not the size of your question.
That is fine when the pile is small or you ask once. It stops being fine when the same incident has you searching a 40 GB window a dozen times, or when an AI agent greps those logs on every turn while it debugs. A purpose-built engine does the reading once, in the background, when nobody is waiting, then keeps a live index that updates as new lines land, so a fresh log entry is searchable again in about a millisecond. At query time there is no file to open and no gigabytes to pull. The result comes back before the scan would have finished its first file.
Same query · same drive · measured
Four ways to search your logs, priced.
Every walker in this table does the same thing: open, read, scan, close, on every query. The only row that does something different is the one that read the pile ahead of time. Measured on one real machine:
| The tool | How it searches your logs | One content query · 4.47M files | vs the index |
|---|---|---|---|
| Interlinked | reads a live index, opens no log files | 16 ms | 1× baseline |
| ripgrep (rg) | every core, skips binaries, but re-reads the bytes each query | 93.8 s | 58,625× slower |
| grep -r | one thread, reads every file line by line, every query | seconds to minutes | re-reads all |
| zgrep / rg -z | the same scan, on compressed rotated logs | grows with the pile | re-reads all |
Methodology: Ryzen 9 9950X3D · 64 GB DDR5 · NVMe · Windows 11 · 4,470,000 files. ripgrep with default settings; the 93.8 s figure is one query in a five-query content set, where the index averaged 58,625× faster across the set. grep -r and zgrep were not separately timed on this volume: they do the same open-read-close scan, so they sit on the same rising line; those cells are the cost model, not a fabricated stopwatch. Full per-tool method in the ripgrep content comparison.
The fix · an index that includes your logs
Read the pile once. Answer forever.
Interlinked keeps a live index of every file on the machine (names and contents), and that includes the log directories your code-search tools skip. It reads the pile once in the background, then answers any later query as a look-up, in the microsecond-to-millisecond range, without opening a single log file. No model, no embeddings, no cloud: just an index that stays on your machine and never leaves it. And because it rides file changes, a line written to a log a moment ago is searchable a moment later.
Keep grep and ripgrep for a single file or a live tail -f. That is exactly what they are built for. The index is for the repeat questions and the whole-machine ones a scan would spend minutes on. It is the same fast floor every AI agent stands on, which is the larger story in the backbone of the LLM economy.
A LOOK-UP IS A LOOK-UP: A WORD, A ROW, A LIST OF LINES. NO MODEL, NO GUESS, NO CLOUD ROUND-TRIP.
Questions people ask
Fast log search, answered.
What is the fastest way to search a large log file?
For a one-off, use ripgrep with a fixed-string pattern (rg -F) and narrow it to the single file or time window you care about. That reads the fewest bytes. For logs you search again and again, an index that reads them once and answers every later query in milliseconds beats any re-scan: measured here at 16 ms against ripgrep's 93.8 s on the same query.
Why is grep so slow on big log files?
grep reads the file's contents on every run and scans a single thread through them, so its cost is the number of bytes: a 40 GB archive takes proportionally longer, every time you ask. ripgrep parallelizes the read across cores and is several times quicker, but it still reads the bytes. The ceiling they share is spelled out in ripgrep vs grep.
Does ripgrep search log files by default?
Often no. ripgrep respects .gitignore and skips hidden files, and log directories are usually ignored, so a plain rg can return nothing while looking like it searched. Pass --no-ignore or the unrestricted -uu to include them.
How do I search compressed (.gz) logs without unzipping them?
Use rg -z, which reads gzip, xz, bzip2 and zstd directly, or the classic zgrep. Both scan the compressed file in place, so you avoid writing gigabytes of decompressed text to disk just to grep it once.
Can an index stay current with logs that are always being written?
Yes. A live index rides file changes, so an appended line is searchable again in about a millisecond (under 30 ms in the worst case) with no re-scan of the file. The index never drifts from what is on disk, and it costs under 1% of the drive to keep.
Stop re-reading your logs
Keep grep. Add a floor under it.
Install once. One signed installer wires up 19 AI clients (Claude Code, Cursor, Copilot, Codex, Windsurf, Zed and the rest) over MCP. No JSON to edit.
Let it index everything. Filenames are searchable immediately; contents fill in behind, including the log and build directories your code-search tools skip. A light background service you will forget is running.
Search once, answer forever. The query that cost ripgrep 93.8 seconds comes back in 16 ms, and a fresh log line is searchable again about a millisecond after it is written.
Everything local is free forever, on all your devices, no card. Hosting starts at $5.99/mo. Download for Windows.