Merge multiple pcap files together, gracefully.
-
Download a precompiled binary from https://github.com/assafmo/joincap/releases
-
Or... Use
go get
:go get -u github.com/assafmo/joincap
-
Or use Ubuntu PPA:
curl -SsL https://assafmo.github.io/ppa/ubuntu/KEY.gpg | sudo apt-key add - sudo curl -SsL -o /etc/apt/sources.list.d/assafmo.list https://assafmo.github.io/ppa/ubuntu/assafmo.list sudo apt update sudo apt install joincap
Usage:
joincap [OPTIONS] InFiles...
Application Options:
-v, --verbose Explain when skipping packets or entire input files
-V, --version Print the version and exit
-w= Sets the output filename. If the name is '-', stdout will be used (default: -)
-c= An integer argument for limiting the pcap size (default: 9223372036854775807)
-p=[micros|nanos] Sets timestamp precision (default: micros)
Help Options:
-h, --help Show this help message
I believe skipping corrupt packets is better than failing the entire merge job.
When using tcpslice
or mergecap
sometimes pcapfix
is needed to fix bad input pcap files.
- One option is to try and run merge (
mergecap
/tcpslice
), if we get errors then runpcapfix
on the bad pcaps and then run merge again.- Adds complexity (run -> check errors -> fix -> rerun)
- (If errors) Demands more resources (
pcapfix
processes) - (If errors) Extends the total run time
- Another option is to run
pcapfix
on the input pcap files and then merge.- Extends the total run time by a lot (read and write each pcap twice instead of once)
- Demands more storage (for the fixed pcaps)
- Demands more resources (
pcapfix
processes)
- We can use
pcapfix
"in memory" with process substitution:mergecap -w out.pcap <(pcapfix -o /dev/stdout 1.pcap) <(pcapfix -o /dev/stdout 2.pcap)
.- Adds complexity (build a complex command line)
- Demands more resources (
pcapfix
processes) - Harder for us to use pathname expansion (e.g.
tcpslice -w out.pcap *.pcap
) - We have to mind the command line character limit (in case of long pathnames)
- Doesn't work for
tcpslice
(seeks the last packets to calculate time ranges - cannot do this with pipes)
Use case | joincap | mergecap v2.4.5 | tcpslice v1.2a3 |
---|---|---|---|
Corrupt input global header | ✔️ | ❌ | ❌ |
Corrupt input packet header | ✔️ | ❌ | ❌ |
Unexpectd EOF (last packet data is truncated) |
✔️ | ✔️ | ✔️ |
Input pcap has no packets (global header is ok, no first packet header) |
✔️ | ✔️ | ❌ |
Input file size is smaller than 24 bytes (global header is truncated) |
✔️ | ✔️ | ❌ |
Input file size is between 24 and 40 bytes (global header is ok, first packet header is truncated) |
✔️ | ❌ | ❌ |
Input file doesn't exists | ✔️ | ❌ | ❌ |
Input file is a directory | ✔️ | ❌ | ❌ |
Input file end is garbage | ✔️ | ✔️ | ❌ |
Input file is gzipped (.pcap.gz) | ✔️ | ✔️ | ❌ |
Use case | Error outputs |
---|---|
Corrupt input global header |
|
Corrupt input packet header |
|
Unexpectd EOF (last packet data is truncated) |
|
Input pcap has no packets (global header is ok, no first packet header) |
|
Input file size is smaller than 24 bytes (global header is truncated) |
|
Input file size is between 24 and 40 bytes (global header is ok, first packet header is truncated) |
|
Input file doesn't exists |
|
Input file is a directory |
|
Input file end is garbage |
|
Input file is gzipped (.pcap.gz) |
|
Use case | How to reproduce |
---|---|
Corrupt input global header |
|
Corrupt input packet header |
|
Unexpectd EOF (last packet data is truncated) |
|
Input pcap has no packets (global header is ok, no first packet header) |
|
Input file size is smaller than 24 bytes (global header is truncated) |
|
Input file size is between 24 and 40 bytes (global header is ok, first packet header is truncated) |
|
Input file doesn't exists |
|
Input file is a directory |
|
Input file end is garbage |
|
Input file is gzipped (.pcap.gz) |
|
Version | Speed | Time | |
---|---|---|---|
mergecap | 3.2.2 | 590MiB/s | 0m5.632s |
tcpslice | 1.2a3 | 838MiB/s | 0m3.666s |
joincap | 0.10.2 | 562MiB/s | 0m5.462s |
- Merging 3 files with total size of 2.99994GiB.
- Running on Linux 5.4.0-21-generic, with Intel(R) Core(TM) i7-9750H CPU @ 2.60GHz (with SSE4.2), with 31765 MB of physical memory, with locale C, with zlib 1.2.11.