https vs http results in different file download

Topic

New Reply

Can someone explain what may be going on here…

I was trying to update a Broadcom Bluetooth driver (BTW_12.0.1.940_win8_10_x64.zip) for a Win10 install on an older laptop. Initially (Aug 8), I used Https to connect to Broadcom’s site but Firefox complained about an untrusted intermediate certificate; so did Chrome, but let me connect. I downloaded the driver file over https on Chrome. I also downloaded the file on a different PC using http with Firefox and noticed a different file size and sha256sum.

No matter what machine/browser/OS I use, whether I use a proxy or not or if I use a different ISP, the file I receive over https is always 77 bytes larger than the one over http. Each version has a consistent sha256sum but different from each other.

I had VirusTotal check both versions of the file and they came back clean. SSLLabs indicates an incomplete server certificate chain for Broadcom, but yesterday FF didn’t complain as much and let me connect without asking to make an exception.

What would explain the files having a different sha256sum and size just because it was downloaded over this https environment? What security/vulnerability is here?

Reply | Quote

Replies

Anytime you use https, it will look for a valid certificate. If it doesn’t find one, it will give you an error message. Getting the error message does not necessarily indicate a security violation; rather, it indicates that the certificate has some error (e.g. perhaps it is expired). If the certificate has some error, then you can’t trust the security of the link, just like you can’t trust it when using http.

As to your question about the different sizes of downloads, I don’t have an answer for that one.

Group "L" (Linux Mint)
with Windows 10 running in a remote session on my file server

Reply | Quote

What would explain the files having a different sha256sum and size just because it was downloaded over this https environment?

The downloaded files have different SHA checksums and file sizes because their attached ‘zone information’ metadata, stored in each file as an alternate data stream (with a stream name of :Zone.Identifier:$DATA), is different.

It’s this ‘zone information’ that causes Windows to display a warning about files that originated ‘elsewhere’, e.g. downloaded from the internet or transferred from another device.

Hope this helps…

EDIT… more info:

You may be able to see this for yourself using Nir Sofer’s AlternateStreamView. For more info, including how to control this behaviour, have a look at Disable downloaded files from being blocked in Windows 10.

Reply | Quote

Thanks Jim, Rick.

The idea of ‘zone information’ metadata is a new one for me so I’ll have to check up on it. I understand http is not secure and trustworthy. Neither is an https connection with an incomplete certificate chain.

So the fact of getting different files over https vs http is not in and of itself so unusual?

Reply | Quote

So the fact of getting different files over https vs http is not in and of itself so unusual?

It’s the same file, only the metadata is different. Here’s a different analogy: You go to the store and buy a dozen eggs. The checkout operator gives you them in a bag. You ask for them to be double-bagged. The differing metadata is the difference between one bag or two… but the eggs remain the same.

For more info about ADS see Windows Alternate Data Streams.

For info about removing ADS using PowerShell, see Alternate Data Streams in NTFS.

Hope this helps…

Reply | Quote

Rick, those were some very good resources on ADS.

I have had a chance now to review them and have checked out my two files (versions). Both SysInternals’ Streams64 and Get-Item from W10 PowerShell indicate there are no ADS related to my files (just the :$DATA). I even unzipped each of them and ran Streams64 with -s but it found no ADS. I am at a loss to know why the same file downloaded from the same site is different based on whether downloaded via https or http.

Reply | Quote

…I am at a loss to know why the same file downloaded from the same site is different based on whether downloaded via https or http.

A previous poster gave this nugget:
“…It’s the same file, only the metadata is different. Here’s a different analogy: You go to the store and buy a dozen eggs. The checkout operator gives you them in a bag. You ask for them to be double-bagged. The differing metadata is the difference between one bag or two… but the eggs remain the same.” — a snippit from a Rick Corbett post
Apparently, the same web site, download site, either single-bags or double-bags the file.

"Take care of thy backups and thy restores shall take care of thee." Ben Franklin, revisted

Reply | Quote

Strange. How can Windows know that a file is not from Zone 0 (i.e. local computer), and has come from ‘elsewhere’ if it doesn’t have an ADS containing at least one stream showing the zone identifier e.g. Zone 3 (internet)? The only reason I can think of is if the filesystem is not NTFS.

Have you considered using something like Beyond Compare, DiffVue or UltraCompare and carrying out a Hex comparison.

Reply | Quote

There is no requirement for HTTP and HTTPS to point at the same file, even though it’s a reasonable expectation.

Could be a caching issue; try downloading again later.

–Scott.

Reply | Quote

Could be a caching issue

That was my thought also. My novice ‘understanding’ is that one of the main differences between HTTP and HTTPS is that the latter doesn’t allow caching. So you might be getting an older version of the file from HTTP.

Lugh.
~
Alienware Aurora R6; Win10 Home x64 1803; Office 365 x32
i7-7700; GeForce GTX 1060; 16GB DDR4 2400; 1TB SSD, 256GB SSD, 4TB HD

Reply | Quote

A hex comparison of the files would be interesting.

Reply | Quote

I am conscious of trying to stay on track here, so bear with me…

Strange. How can Windows know that a file is not from Zone 0 (i.e. local computer), and has come from ‘elsewhere’ if it doesn’t have an ADS containing at least one stream showing the zone identifier e.g. Zone 3 (internet)? The only reason I can think of is if the filesystem is not NTFS.

You’re correct. I have different machines available to me depending on where I am during the day. When I first tried to test the ADS issue, I had a Win7 PC and the -stream option was bugging out. I guess it doesn’t work with Win7. So I used a Win10 install within a VM. The use of the VM apparently had stripped out the zone identifier streams (all ADS I guess). When doing the ADS checks on the Win10 laptop itself, those zone identifier streams were present. They were the same for all the files and no other ADS streams were there.

Have you considered using something like Beyond Compare, DiffVue or UltraCompare and carrying out a Hex comparison.

I’ll check those out to try and find what is different between the two versions of the zip files.

There is no requirement for HTTP and HTTPS to point at the same file, even though it’s a reasonable expectation.

Could be a caching issue; try downloading again later.

It was my assumption that they would point to the same file, just without/with encrypted communications. Re: caching, the differences have been consistent for at least a week.

The original Firefox certificate issue and sslLabs comment of an incomplete certificate chain raised my suspicions, but the fact that SHA checksums and file sizes were different (ADS now ruled out as a possible cause) stoked the fire. It is hard to know what to have trust in.

Reply | Quote

After doing the Hex comparison (I’d never done that before), the differences are accounted for by 77 instances of various pointers to the broadcom website. The https version included the extra “s” character 77 times. See attached screen grab.
45584-BC_hex_pic4

Reply | Quote

After doing the Hex comparison (I’d never done that before), the differences are accounted for by 77 instances of various pointers to the broadcom website. The https version included the extra “s” character 77 times.

The file you are querying with BeyondCompare is the ‘Broadcom WIDCOMM Bluetooth Driver 12.0.1.940’ so it’s no surprise that there are references to the Broadcom website within the HEX comparison of the driver ZIP file.

As the only apparent differences between the two downloads are the inclusion of ‘s’ (as in HTTPS) and VirusTotal comes back clean on both files I don’t think you have anything to worry about.

Hope this helps…

Reply | Quote

It’s very handy the way the Hex Compare program keeps everything aligned between the two windows so that you can easily see what is missing on the left.

Yes. I did like the simplicity of Beyond Compare (using default settings), of the three suggested by Rick. The others didn’t do such a clean job and I didn’t have time to dig into the various options/settings.

The file you are querying with BeyondCompare is the ‘Broadcom WIDCOMM Bluetooth Driver 12.0.1.940’ so it’s no surprise that there are references to the Broadcom website within the HEX comparison of the driver ZIP file.

As the only apparent differences between the two downloads are the inclusion of ‘s’ (as in HTTPS) and VirusTotal comes back clean on both files I don’t think you have anything to worry about.

Hope this helps…

That gets back to what I was originally concerned with. The hex comparison is a new tool I can use in future. It’s been a great learning exercise. Thanks to everyone.

Reply | Quote

Very interesting comparison. You’ll note that the hexadecimal numbers in the left column of each window show the total number of bytes up to that point. Each time the “s” is missing on the left side, the total number of bytes shown in the next row is one less with the left window vs the right window.

The hexadecimal digits are 0 1 2 3 4 5 6 7 8 9 A B C D E F. That’s why, when you have eight bytes, you count from 28 to 30, whereas when you have seven bytes, you count from 28 to 2F — …25, 26, 27, 28, 29, 2A, 2B, 2C, 2D, 2E, 2F, 30, 31, 32…

By comparison, the decimal digits are 0 1 2 3 4 5 6 7 8 9. Counting in decimal would be: …25, 26, 27, 28, 29, 30, 31, 32…

Decimal is base 10, whereas hexadecimal is base 16.

It’s very handy the way the Hex Compare program keeps everything aligned between the two windows so that you can easily see what is missing on the left.

Group "L" (Linux Mint)
with Windows 10 running in a remote session on my file server

Reply | Quote