Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Different Youtube classification on little-endian x big-endian machine #1475

Closed
1 task done
viniciussn opened this issue Mar 4, 2022 · 12 comments
Closed
1 task done

Different Youtube classification on little-endian x big-endian machine #1475

viniciussn opened this issue Mar 4, 2022 · 12 comments
Labels
bug

Comments

@viniciussn
Copy link
Contributor

Hello,

Running ndpiReader with a Youtube traffic pcap is giving me different outputs on a little-endian and a big-endian machine with the latest commit (95a3d4f).

Little-endian output 
root@machine# ./ndpiReader -i youtube.pcap  

-----------------------------------------------------------
* NOTE: This is demo app to show *some* nDPI features.
* In this demo we have implemented only some basic features
* just to show you what you can do with the library. Feel 
* free to extend it and send us the patches for inclusion
------------------------------------------------------------

Using nDPI (4.3.0-3520-95a3d4ff) [1 thread(s)]
Using libgcrypt version 1.8.6internal
Reading packets from pcap file /home/vinicius/Downloads/subir/youtube.pcap...
Running thread 0...

nDPI Memory statistics:
	nDPI Memory (once):      237.23 KB    
	Flow Memory (per flow):  688 B        
	Actual Memory:           25.00 MB     
	Peak Memory:             25.00 MB     
	Setup Time:              37 msec
	Packet Processing Time:  18 msec

Traffic statistics:
	Ethernet bytes:        21308413      (includes ethernet CRC/IFC/trailer)
	Discarded bytes:       0            
	IP packets:            19999         of 19999 packets total
	IP bytes:              20828437      (avg pkt size 1041 bytes)
	Unique flows:          47           
	TCP Packets:           2857         
	UDP Packets:           17140        
	VLAN Packets:          0            
	MPLS Packets:          0            
	PPPoE Packets:         0            
	Fragmented Packets:    0            
	Max Packet size:       5704         
	Packet Len < 64:       5611         
	Packet Len 64-128:     151          
	Packet Len 128-256:    59           
	Packet Len 256-1024:   199          
	Packet Len 1024-1500:  13562        
	Packet Len > 1500:     417          
	nDPI throughput:       1.09 M pps / 8.64 Gb/sec
	Analysis begin:        04/Mar/2022 11:22:52
	Analysis end:          04/Mar/2022 11:23:22
	Traffic throughput:    657.37 pps / 5.34 Mb/sec
	Traffic duration:      30.423 sec
	Guessed flow protos:   10           
	DPI Packets (TCP):     249           (8.03 pkts/flow)
	DPI Packets (UDP):     16            (1.07 pkts/flow)
	DPI Packets (other):   1             (1.00 pkts/flow)
	Confidence: Match by port 6             (flows)
	Confidence: Match by IP 4             (flows)
	Confidence: DPI        37            (flows)


Detected protocols:
	DNS                  packets: 2             bytes: 190           flows: 1            
	HTTP                 packets: 20            bytes: 1320          flows: 3            
	OCSP                 packets: 102           bytes: 30637         flows: 6            
	ICMP                 packets: 2             bytes: 196           flows: 1            
	SSH                  packets: 70            bytes: 9884          flows: 2            
	YouTube              packets: 19098         bytes: 20454926      flows: 14           
	Google               packets: 585           bytes: 298742        flows: 13           
	TeamViewer           packets: 5             bytes: 348           flows: 1            
	Cloudflare           packets: 10            bytes: 570           flows: 1            
	GoogleServices       packets: 87            bytes: 30436         flows: 2            
	AmazonAWS            packets: 6             bytes: 396           flows: 1            
	GoogleCloud          packets: 12            bytes: 792           flows: 2            


Protocol statistics:
	Safe                         30637 bytes
	Acceptable                  342874 bytes
	Fun                       20454926 bytes
Big-endian output 
root@machine# ./ndpiReader -i youtube.pcap 

-----------------------------------------------------------
* NOTE: This is demo app to show *some* nDPI features.
* In this demo we have implemented only some basic features
* just to show you what you can do with the library. Feel 
* free to extend it and send us the patches for inclusion
------------------------------------------------------------

Using nDPI (4.3.0-3520-95a3d4ff) [1 thread(s)]
Using libgcrypt version 1.8.6internal
Reading packets from pcap file youtube.pcap...
Running thread 0...

nDPI Memory statistics:
	nDPI Memory (once):      223.71 KB    
	Flow Memory (per flow):  632 B        
	Actual Memory:           23.26 MB     
	Peak Memory:             23.26 MB     
	Setup Time:              769 msec
	Packet Processing Time:  729 msec

Traffic statistics:
	Ethernet bytes:        21308413      (includes ethernet CRC/IFC/trailer)
	Discarded bytes:       0            
	IP packets:            19999         of 19999 packets total
	IP bytes:              20828437      (avg pkt size 1041 bytes)
	Unique flows:          47           
	TCP Packets:           2857         
	UDP Packets:           17140        
	VLAN Packets:          0            
	MPLS Packets:          0            
	PPPoE Packets:         0            
	Fragmented Packets:    0            
	Max Packet size:       5704         
	Packet Len < 64:       5611         
	Packet Len 64-128:     151          
	Packet Len 128-256:    59           
	Packet Len 256-1024:   199          
	Packet Len 1024-1500:  13562        
	Packet Len > 1500:     417          
	nDPI throughput:       27.42 K pps / 222.87 Mb/sec
	Analysis begin:        04/Mar/2022 14:22:52
	Analysis end:          04/Mar/2022 14:23:22
	Traffic throughput:    657.37 pps / 5.34 Mb/sec
	Traffic duration:      30.423 sec
	Guessed flow protos:   10           
	DPI Packets (TCP):     249           (8.03 pkts/flow)
	DPI Packets (UDP):     16            (1.07 pkts/flow)
	DPI Packets (other):   1             (1.00 pkts/flow)
	Confidence: Match by port 6             (flows)
	Confidence: Match by IP 4             (flows)
	Confidence: DPI        37            (flows)


Detected protocols:
	DNS                  packets: 2             bytes: 190           flows: 1            
	HTTP                 packets: 20            bytes: 1320          flows: 3            
	OCSP                 packets: 102           bytes: 30637         flows: 6            
	ICMP                 packets: 2             bytes: 196           flows: 1            
	SSH                  packets: 70            bytes: 9884          flows: 2            
	YouTube              packets: 2247          bytes: 2605867       flows: 8            
	Google               packets: 1972          bytes: 1561744       flows: 18           
	TeamViewer           packets: 5             bytes: 348           flows: 1            
	QUIC                 packets: 15510         bytes: 16605433      flows: 2            
	Cloudflare           packets: 10            bytes: 570           flows: 1            
	GoogleServices       packets: 41            bytes: 11060         flows: 1            
	AmazonAWS            packets: 6             bytes: 396           flows: 1            
	GoogleCloud          packets: 12            bytes: 792           flows: 2            


Protocol statistics:
	Safe                         30637 bytes
	Acceptable                18191933 bytes
	Fun                        2605867 bytes
  • Architectures: x86-64 and MIPS
  • nDPI version or commit hash: 95a3d4f.
  • nDPI compilation flags used: None. Using the new internal crypto code.
  • The reported bug is reproducible using ndpiReader.

Command used: ndpiReader -i youtube.pcap

Example: youtube.pcap
@viniciussn viniciussn added the bug label Mar 4, 2022
@utoni
Copy link
Collaborator

utoni commented Mar 4, 2022

Possible duplicate of #1312.

@viniciussn
Copy link
Contributor Author

Do you agree with @IvanNardi?

these issues seem to be related to ndpiReader application, not libndpi.so itself

Because I'm using libndpi in another application and this issue is also reproducible there.

@IvanNardi
Copy link
Collaborator

Do you agree with @IvanNardi?

these issues seem to be related to ndpiReader application, not libndpi.so itself

Because I'm using libndpi in another application and this issue is also reproducible there.

Please note that comment has been made before the addition of the internal crypto code.
I would not be surprise at all if that code has some issues on big-endian machines.

@IvanNardi
Copy link
Collaborator

Definitely some endianess issue on internal crypto library: https://github.com/ntop/nDPI/runs/5424718108?check_suite_focus=true all QUIC tests are failing on big-endian arch

@vel21ripn
Copy link
Contributor

Does the error only appear when internal gcrypt is used?
At the time of the adoption of #1444 there was no testing on s390 with built-in libgcrypt?

@viniciussn
Copy link
Contributor Author

I'm building the latest version with host libgcrypt and will post the results ASAP.

@viniciussn
Copy link
Contributor Author

viniciussn commented Mar 4, 2022 •
edited
Loading

Here is the output of ndpiReader with libgcrypt 1.9.3, running on a big-endian machine:

output 
root@machine# ./ndpiReader -i youtube.pcap 

-----------------------------------------------------------
* NOTE: This is demo app to show *some* nDPI features.
* In this demo we have implemented only some basic features
* just to show you what you can do with the library. Feel 
* free to extend it and send us the patches for inclusion
------------------------------------------------------------

Using nDPI (4.3.0-3520-95a3d4ff) [1 thread(s)]
Using libgcrypt version 1.9.3
Reading packets from pcap file youtube.pcap...
Running thread 0...

nDPI Memory statistics:
	nDPI Memory (once):      223.71 KB    
	Flow Memory (per flow):  632 B        
	Actual Memory:           23.21 MB     
	Peak Memory:             23.21 MB     
	Setup Time:              796 msec
	Packet Processing Time:  784 msec

Traffic statistics:
	Ethernet bytes:        21308413      (includes ethernet CRC/IFC/trailer)
	Discarded bytes:       0            
	IP packets:            19999         of 19999 packets total
	IP bytes:              20828437      (avg pkt size 1041 bytes)
	Unique flows:          47           
	TCP Packets:           2857         
	UDP Packets:           17140        
	VLAN Packets:          0            
	MPLS Packets:          0            
	PPPoE Packets:         0            
	Fragmented Packets:    0            
	Max Packet size:       5704         
	Packet Len < 64:       5611         
	Packet Len 64-128:     151          
	Packet Len 128-256:    59           
	Packet Len 256-1024:   199          
	Packet Len 1024-1500:  13562        
	Packet Len > 1500:     417          
	nDPI throughput:       25.50 K pps / 207.26 Mb/sec
	Analysis begin:        04/Mar/2022 14:22:52
	Analysis end:          04/Mar/2022 14:23:22
	Traffic throughput:    657.37 pps / 5.34 Mb/sec
	Traffic duration:      30.423 sec
	Guessed flow protos:   10           
	DPI Packets (TCP):     249           (8.03 pkts/flow)
	DPI Packets (UDP):     16            (1.07 pkts/flow)
	DPI Packets (other):   1             (1.00 pkts/flow)
	Confidence: Match by port 6             (flows)
	Confidence: Match by IP 4             (flows)
	Confidence: DPI        37            (flows)


Detected protocols:
	DNS                  packets: 2             bytes: 190           flows: 1            
	HTTP                 packets: 20            bytes: 1320          flows: 3            
	OCSP                 packets: 102           bytes: 30637         flows: 6            
	ICMP                 packets: 2             bytes: 196           flows: 1            
	SSH                  packets: 70            bytes: 9884          flows: 2            
	YouTube              packets: 19098         bytes: 20454926      flows: 14           
	Google               packets: 585           bytes: 298742        flows: 13           
	TeamViewer           packets: 5             bytes: 348           flows: 1            
	Cloudflare           packets: 10            bytes: 570           flows: 1            
	GoogleServices       packets: 87            bytes: 30436         flows: 2            
	AmazonAWS            packets: 6             bytes: 396           flows: 1            
	GoogleCloud          packets: 12            bytes: 792           flows: 2            


Protocol statistics:
	Safe                         30637 bytes
	Acceptable                  342874 bytes
	Fun                       20454926 bytes

@IvanNardi
Copy link
Collaborator

I confirm: QUIC dissection is fine with external libgcrypt on BE machines: https://github.com/ntop/nDPI/runs/5426850761?check_suite_focus=true

@vel21ripn
Copy link
Contributor

vel21ripn commented Mar 4, 2022 •
edited
Loading

We need to check the functionality of the "tests/performance/gcrypt" utilities on the s390.
I made a test system on s390/qemu/ubuntu-18.
I'll try to figure out where the error is.

@vel21ripn
Copy link
Contributor

See #1478
I see strange test errors on s390 (not related to Endian issues).
1kxun.pcap: Number of packets and bytes do not match.
ethernetIP.pcap: category does not match.
KakaoTalk_chat.pcap: errors related to the ICMP protocol.
quic_interop_V.pcapng: errors related to the ICMP protocol.
skype_no_unknown.pcap,teams.pcap,zoom.pcap: The number of packets and bytes does not match.

@IvanNardi
Copy link
Collaborator

EthernetIP should be fixed in #1477
@lnslbrty , it seems that ICMP checksum calculation (or comparison) is wrong on BE machines. Could you take a look, please?

The other issues (already mentioned in #1312 (comment)) seems to be related to ndpiReader code.

@viniciussn , could you confirm that this specific issue can be close, please?

@viniciussn
Copy link
Contributor Author

Hello,

Now it is working as expected. Output on a big-endian machine:

output 
root@machine# ./ndpiReader -i youtube.pcap 

-----------------------------------------------------------
* NOTE: This is demo app to show *some* nDPI features.
* In this demo we have implemented only some basic features
* just to show you what you can do with the library. Feel 
* free to extend it and send us the patches for inclusion
------------------------------------------------------------

Using nDPI (4.3.0-3523-c345b3c7) [1 thread(s)]
Using libgcrypt version 1.8.6internal
Reading packets from pcap file youtube.pcap...
Running thread 0...

nDPI Memory statistics:
	nDPI Memory (once):      223.71 KB    
	Flow Memory (per flow):  632 B        
	Actual Memory:           23.26 MB     
	Peak Memory:             23.26 MB     
	Setup Time:              753 msec
	Packet Processing Time:  737 msec

Traffic statistics:
	Ethernet bytes:        21308413      (includes ethernet CRC/IFC/trailer)
	Discarded bytes:       0            
	IP packets:            19999         of 19999 packets total
	IP bytes:              20828437      (avg pkt size 1041 bytes)
	Unique flows:          47           
	TCP Packets:           2857         
	UDP Packets:           17140        
	VLAN Packets:          0            
	MPLS Packets:          0            
	PPPoE Packets:         0            
	Fragmented Packets:    0            
	Max Packet size:       5704         
	Packet Len < 64:       5611         
	Packet Len 64-128:     151          
	Packet Len 128-256:    59           
	Packet Len 256-1024:   199          
	Packet Len 1024-1500:  13562        
	Packet Len > 1500:     417          
	nDPI throughput:       27.11 K pps / 220.36 Mb/sec
	Analysis begin:        04/Mar/2022 14:22:52
	Analysis end:          04/Mar/2022 14:23:22
	Traffic throughput:    657.37 pps / 5.34 Mb/sec
	Traffic duration:      30.423 sec
	Guessed flow protos:   10           
	DPI Packets (TCP):     249           (8.03 pkts/flow)
	DPI Packets (UDP):     16            (1.07 pkts/flow)
	DPI Packets (other):   1             (1.00 pkts/flow)
	Confidence: Match by port 6             (flows)
	Confidence: Match by IP 4             (flows)
	Confidence: DPI        37            (flows)


Detected protocols:
	DNS                  packets: 2             bytes: 190           flows: 1            
	HTTP                 packets: 20            bytes: 1320          flows: 3            
	OCSP                 packets: 102           bytes: 30637         flows: 6            
	ICMP                 packets: 2             bytes: 196           flows: 1            
	SSH                  packets: 70            bytes: 9884          flows: 2            
	YouTube              packets: 19098         bytes: 20454926      flows: 14           
	Google               packets: 585           bytes: 298742        flows: 13           
	TeamViewer           packets: 5             bytes: 348           flows: 1            
	Cloudflare           packets: 10            bytes: 570           flows: 1            
	GoogleServices       packets: 87            bytes: 30436         flows: 2            
	AmazonAWS            packets: 6             bytes: 396           flows: 1            
	GoogleCloud          packets: 12            bytes: 792           flows: 2            


Protocol statistics:
	Safe                         30637 bytes
	Acceptable                  342874 bytes
	Fun                       20454926 bytes

Thank you guys for fixing it so fast.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@utoni @vel21ripn @IvanNardi @viniciussn