[ESP8266HTTPClient] Updates from HTTPS servers fail before transferring new binary begins

[pacau999]

http update not working on this arduino-core version

HTTP_UPDATE_FAILD Error (-106): Verify Bin Header Failed;
I just instaled arduino core 2.7.4 and it worked.

NodeMCU 1.0, Arduino 1.8.15, code

httpUpdateSecure

[earlephilhower]

Please run with full debugging enabled and attach the logs. That can make pinpointing things much simpler for everyone.

[frankygoop]

I got the same error with Core 3.0.0. With Core 2.7.4 works fine.

[d-a-v]

You will help us if you consider the previous message:

Please run with full debugging enabled and attach the logs. That can make pinpointing things much simpler for everyone.

[frankygoop]

I will be glad to help. Would you indicate how to get the log?

[d-a-v]

Thanks !
In the arduino IDE, it's tools>Debug Port and tools>Debug Level

[frankygoop]

Hello, What level of debug do you require?

[d-a-v]

I think the one before the last would give all details.

[frankygoop]

This is all I could catch. I had to reduce the level of debug because it was scrolling too fast.
I am downloading the bin file from a private Github. I had to add Bearer header into the library to make it work. I dont know any method to add custom headers to the HttpUpdate.

[HTTP-Client][begin] url: https://api.github.com/repos/...............bin
[HTTP-Client][begin] host: api.github.com port: 443 url: /repos/............bin
[HTTP-Client][sendRequest] type: 'GET' redirCount: 0
BSSL:_connectSSL: start connection
BSSL:CERT: 30 82 05 05 30 82 04 ab a0 03 02 01 02 02 10 01 68 d5 75 f1 ce 87 28 ad 95 a8 f1 1e f1 59 8b 30 0a 06 08 2a 86 48 ce 3d 04 03 02 30 67 31 0b 30 09 06 03 55 04 06 13 02 55 53 31 17 30 15 06 03 55 04 0a 13 0e 44 69 67 69 43 65 72 74 2c 20 49 6e 63 2e 31 3f 30 3d 06 03 55 04 03 13 36 44 69 67 69 43 65 72 74 20 48 69 67 68 20 41 73 73 75 72 61 6e 63 65 20 54 4c 53 20 48 79 62 72 69 64 20 45 43 43 20 53 48 41 32 35 36 20 32 30 32 30 20 43 41 31 30 1e 17 0d 32 31 30 33 32 35 30 30 30 30 30 30 5a 17 0d 32 32 30 33 33 30 32 33 35 39 35 39 5a 30 68 31 0b 30 09 06 03 55 04 06 13 02 55 53 31 13 30 11 06 03 55 04 08 13 0a 43 61 6c 69 66 6f 72 6e 69 61 31 16 30 14 06 03 55 04 07 13 0d 53 61 6e 20 46 72 61 6e 63 69 73 63 6f 31 15 30 13 06 03 55 04 0a 13 0c 47 69 74 48 75 
BSSL:CERT: 62 2c 20 49 6e 63 2e 31 15 30 13 06 03 55 04 03 0c 0c 2a 2e 67 69 74 68 75 62 2e 63 6f 6d 30 59 30 13 06 07 2a 86 48 ce 3d 02 01 06 08 2a 86 48 ce 3d 03 01 07 03 42 00 04 94 bf 7f fb 44 e2 75 32 00 2c f7 d4 fe d8 92 ea 92 3c cc 02 24 ad 4e 29 a2 15 25 75 57 34 6d be 8e dc 50 11 97 77 27 a8 80 ca f5 2a 05 eb 9c f7 c6 24 7a 5e c9 c5 4b c1 74 3b 2f dc 85 74 7f 91 a3 82 03 36 30 82 03 32 30 1f 06 03 55 1d 23 04 18 30 16 80 14 50 61 a6 a0 d2 35 c4 11 2a 20 8d 1f 0f ac 42 f0 cd 29 cf 4b 30 1d 06 03 55 1d 0e 04 16 04 14 cf cb 0c eb ee 3d 71 24 f8 7e 96 5a 71 a6 4a 9b d5 dc c6 ce 30 23 06 03 55 1d 11 04 1c 30 1a 82 0c 2a 2e 67 69 74 68 75 62 2e 63 6f 6d 82 0a 67 69 74 68 75 62 2e 63 6f 6d 30 0e 06 03 55 1d 0f 01 01 ff 04 04 03 02 07 80 30 1d 06 03 55 1d 25 04 16 30 
BSSL:CERT: 14 06 08 2b 06 01 05 05 07 03 01 06 08 2b 06 01 05 05 07 03 02 30 81 b1 06 03 55 1d 1f 04 81 a9 30 81 a6 30 51 a0 4f a0 4d 86 4b 68 74 74 70 3a 2f 2f 63 72 6c 33 2e 64 69 67 69 63 65 72 74 2e 63 6f 6d 2f 44 69 67 69 43 65 72 74 48 69 67 68 41 73 73 75 72 61 6e 63 65 54 4c 53 48 79 62 72 69 64 45 43 43 53 48 41 32 35 36 32 30 32 30 43 41 31 2e 63 72 6c 30 51 a0 4f a0 4d 86 4b 68 74 74 70 3a 2f 2f 63 72 6c 34 2e 64 69 67 69 63 65 72 74 2e 63 6f 6d 2f 44 69 67 69 43 65 72 74 48 69 67 68 41 73 73 75 72 61 6e 63 65 54 4c 53 48 79 62 72 69 64 45 43 43 53 48 41 32 35 36 32 30 32 30 43 41 31 2e 63 72 6c 30 3e 06 03 55 1d 20 04 37 30 35 30 33 06 06 67 81 0c 01 02 02 30 29 30 27 06 08 2b 06 01 05 05 07 02 01 16 1b 68 74 74 70 3a 2f 2f 77 77 77 2e 64 69 67 69 63 65 72 
BSSL:CERT: 74 2e 63 6f 6d 2f 43 50 53 30 81 92 06 08 2b 06 01 05 05 07 01 01 04 81 85 30 81 82 30 24 06 08 2b 06 01 05 05 07 30 01 86 18 68 74 74 70 3a 2f 2f 6f 63 73 70 2e 64 69 67 69 63 65 72 74 2e 63 6f 6d 30 5a 06 08 2b 06 01 05 05 07 30 02 86 4e 68 74 74 70 3a 2f 2f 63 61 63 65 72 74 73 2e 64 69 67 69 63 65 72 74 2e 63 6f 6d 2f 44 69 67 69 43 65 72 74 48 69 67 68 41 73 73 75 72 61 6e 63 65 54 4c 53 48 79 62 72 69 64 45 43 43 53 48 41 32 35 36 32 30 32 30 43 41 31 2e 63 72 74 30 0c 06 03 55 1d 13 01 01 ff 04 02 30 00 30 82 01 03 06 0a 2b 06 01 04 01 d6 79 02 04 02 04 81 f4 04 81 f1 00 ef 00 76 00 29 79 be f0 9e 39 39 21 f0 56 73 9f 63 a5 77 e5 be 57 7d 9c 60 0a f8 f9 4d 5d 26 5c 25 5d c7 84 00 00 01 78 6a bd f9 54 00 00 04 03 00 47 30 45 02 20 04 c9 f0 35 07 92 20 
BSSL:CERT: 80 05 4a ea 86 c0 5c ff 58 ac 53 cd c3 a5 16 03 47 20 e7 e6 49 78 d1 d7 fa 02 21 00 81 fa c9 e9 a9 88 a2 8c 25 0a 68 04 d8 1a 5c e9 1e 0f 73 b9 e4 88 61 66 16 da b7 f6 81 d5 5e 84 00 75 00 22 45 45 07 59 55 24 56 96 3f a1 2f f1 f7 6d 86 e0 23 26 63 ad c0 4b 7f 5d c6 83 5c 6e e2 0f 02 00 00 01 78 6a bd f9 ad 00 00 04 03 00 46 30 44 02 20 5b 0d 6a 85 67 0b 5b 32 38 e2 46 de a3 2b 15 50 ed 0c 82 19 bb 25 92 92 35 55 29 e5 ab 59 56 60 02 20 41 e3 db ac bb 6c 55 cf df fb b5 2a 8d 73 ed f0 18 60 c6 d7 6f 9b f4 f3 f4 7c 44 a8 b6 ec a5 d7 30 0a 06 08 2a 86 48 ce 3d 04 03 02 03 48 00 30 45 02 21 00 b9 a1 93 26 e3 64 bb b2 c7 c4 0a 22 81 63 88 6f 0d e4 3b 34 da 67 1a 14 67 ef cd 6d 59 a2 85 cb 02 20 22 f3 f9 2b 24 ab fa 84 e6 e8 f2 96 7b 81 e0 3b b6 86 f0 ba ee 51 57 
BSSL:CERT: de ca c1 08 93 c3 f0 f5 f6 
BSSL:Connected!
[HTTP-Client] connected to api.github.com:443
[HTTP-Client] sending request header
-----
GET /repos/.................bin HTTP/1.0
Host: api.github.com
User-Agent: ESP8266-http-Update
Connection: close
x-ESP8266-Chip-ID: 7904961
x-ESP8266-STA-MAC: DC:4F:22:78:9E:C1
x-ESP8266-AP-MAC: DE:4F:22:78:9E:C1
x-ESP8266-free-space: 2641920
x-ESP8266-sketch-size: 500768
x-ESP8266-sketch-md5: c27965.............26992
x-ESP8266-chip-size: 4194304
x-ESP8266-sdk-version: 2.2.2-dev(38a443e)
Accept: application/vnd.github.v3.raw
authorization: Bearer gh.......................HH
x-ESP8266-mode: sketch
Content-Length: 0

-----
[HTTP-Client][handleHeaderResponse] RX: 'HTTP/1.1 200 OK
'
[HTTP-Client][handleHeaderResponse] RX: 'Server: GitHub.com
'
[HTTP-Client][handleHeaderResponse] RX: 'Date: Sat, 29 May 2021 00:48:09 GMT
'
[HTTP-Client][handleHeaderResponse] RX: 'Content-Type: application/vnd.github.v3.raw; charset=utf-8
'
[HTTP-Client][handleHeaderResponse] RX: 'Content-Length: 483232
'
[HTTP-Client][handleHeaderResponse] RX: 'Cache-Control: private, max-age=60, s-maxage=60
'
[HTTP-Client][handleHeaderResponse] RX: 'Vary: Accept, Authorization, Cookie, X-GitHub-OTP
'
[HTTP-Client][handleHeaderResponse] RX: 'ETag: "932df99358f7ff14ef80902e39234b700e76c77f"
'
[HTTP-Client][handleHeaderResponse] RX: 'Last-Modified: Fri, 28 May 2021 21:45:58 GMT
'
[HTTP-Client][handleHeaderResponse] RX: 'X-OAuth-Scopes: read:org, repo
'
[HTTP-Client][handleHeaderResponse] RX: 'X-Accepted-OAuth-Scopes: 
'
[HTTP-Client][handleHeaderResponse] RX: 'X-GitHub-Media-Type: github.v3; param=raw
'
[HTTP-Client][handleHeaderResponse] RX: 'X-RateLimit-Limit: 5000
'
[HTTP-Client][handleHeaderResponse] RX: 'X-RateLimit-Remaining: 4996
'
[HTTP-Client][handleHeaderResponse] RX: 'X-RateLimit-Reset: 1622252269
'
[HTTP-Client][handleHeaderResponse] RX: 'X-RateLimit-Used: 4
'
[HTTP-Client][handleHeaderResponse] RX: 'X-RateLimit-Resource: core
'
[HTTP-Client][handleHeaderResponse] RX: 'Access-Control-Expose-Headers: ETag, Link, Location, Retry-After, X-GitHub-OTP, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Used, X-RateLimit-Resource, X-RateLimit-Reset, X-OAuth-Scopes, X-Accepted-OAuth-Scopes, X-Poll-Interval, X-GitHub-Media-Type, Deprecation, Sunset
'
[HTTP-Client][handleHeaderResponse] RX: 'Access-Control-Allow-Origin: *
'
[HTTP-Client][handleHeaderResponse] RX: 'Strict-Transport-Security: max-age=31536000; includeSubdomains; preload
'
[HTTP-Client][handleHeaderResponse] RX: 'X-Frame-Options: deny
'
[HTTP-Client][handleHeaderResponse] RX: 'X-Content-Type-Options: nosniff
'
[HTTP-Client][handleHeaderResponse] RX: 'X-XSS-Protection: 0
'
[HTTP-Client][handleHeaderResponse] RX: 'Referrer-Policy: origin-when-cross-origin, strict-origin-when-cross-origin
'
[HTTP-Client][handleHeaderResponse] RX: 'Content-Security-Policy: default-src 'none'
'
[HTTP-Client][handleHeaderResponse] RX: 'Vary: Accept-Encoding, Accept, X-Requested-With
'
[HTTP-Client][handleHeaderResponse] RX: 'X-GitHub-Request-Id: CE7D:3158:C4F842:17C37CE:60B18F47
'
[HTTP-Client][handleHeaderResponse] RX: 'connection: close
'
[HTTP-Client][handleHeaderResponse] RX: '
'
[HTTP-Client][handleHeaderResponse] code: 200
[HTTP-Client][handleHeaderResponse] size: 483232
BSSL:peekBytes: Not connected
[HTTP-Client][end] tcp is closed
HTTP_UPDATE_FAILD Error (-106): Verify Bin Header Failed

[earlephilhower]

I just ran a test of the httpUpdate example against my local server and found no issue. This may be related to the actual binary you're trying to download. Can someone please post a public link we can run against? Right now, I'm unable to repro anything:

10:42:02.509 -> [SETUP] WAIT 2...
10:42:03.503 -> [SETUP] WAIT 1...
10:42:05.856 -> CALLBACK:  HTTP update process started
10:42:05.956 -> CALLBACK:  HTTP update process at 0 of 270576 bytes...
10:42:05.989 -> CALLBACK:  HTTP update process at 0 of 270576 bytes...
10:42:06.022 -> CALLBACK:  HTTP update process at 4096 of 270576 bytes...
10:42:06.088 -> CALLBACK:  HTTP update process at 8192 of 270576 bytes...
10:42:06.187 -> CALLBACK:  HTTP update process at 12288 of 270576 bytes...
...
10:42:10.462 -> CALLBACK:  HTTP update process at 270576 of 270576 bytes...
10:42:10.462 -> CALLBACK:  HTTP update process at 270576 of 270576 bytes...
10:42:10.462 -> CALLBACK:  HTTP update process finished
10:42:10.561 -> 
10:42:10.561 ->  ets Jan  8 2013,rst cause:2, boot mode:(3,6)
10:42:10.561 -> 
10:42:10.561 -> load 0x4010f000, len 3460, room 16 
10:42:10.561 -> tail 4
10:42:10.561 -> chksum 0xcc
10:42:10.561 -> load 0x3fff20b8, len 40, room 4 
10:42:10.561 -> tail 4
10:42:10.561 -> chksum 0xc9
10:42:10.561 -> csum 0xc9
10:42:10.561 -> v0004b1e0
10:42:10.561 -> @cp:B0
10:42:13.975 -> ld
10:42:14.008 -> 
10:42:14.008 -> 
10:42:14.008 -> ** pgm_read_float_unaligned() test **
10:42:14.008 -> specimen: 3.141593
10:42:14.008 -> readout:  3.141593

[frankygoop]

I suggest to try to do the test from a private github repository. The bin file I am downloading is the same that works fine with core 2.7.4.

[earlephilhower]

@frankygoop sorry, but I don't have any private repos as the maintainers don't use GH professionally.

If you or someone else has, say, a public blink.binthat you could repro the failure on that would be fine. Especially if the bin is an older one and was built in some way different than we're using locally (since it's reporting a format error).

[zhfei1979]

Got the same issue with 3.0.0, works fine with 2.7.4.
Issue can be reproduced with:
BearSSL::WiFiClientSecure client;
client.setFingerprint(cfgFP);
ESPhttpUpdate.update(client, urlHTTPS);

=============== ESP8266httpUpdate.cpp handleUpdate==========
WiFiClient * tcp = http.getStreamPtr();

....
uint8_t buf[4];
uint8_t x = tcp->peekBytes(&buf[0], 4);
DEBUG_HTTP_UPDATE("[httpUpdate] peekBytes x：%d\n", x);
if(x != 4) {
DEBUG_HTTP_UPDATE("[httpUpdate] peekBytes magic header failed\n");
_setLastError(HTTP_UE_BIN_VERIFY_HEADER_FAILED);
http.end();
return HTTP_UPDATE_FAILED;
}

The above codes prints x=0 causes the "Verify Bin Header Failed" error:

[httpUpdate] Header read fin.
[httpUpdate] Server header:
[httpUpdate] - code: 200
[httpUpdate] - len: 466352
[httpUpdate] ESP8266 info:
[httpUpdate] - free Space: 581632
[httpUpdate] - current Sketch Size: 466352
[httpUpdate] runUpdate flash...
[httpUpdate] peekBytes x：0
[httpUpdate] peekBytes magic header failed
UpdateByOTA: Verify Bin Header Failed!, 8261
state: 5 -> 0 (0)
rm 0
del if0
usl

[earlephilhower]

Again, folks, we need a public example that fails w/3.0 but passes w/the prior release. I've tried my own way several times and things "just work" so there's nothing we can do w/o a failing public example.

If it's failing but you've got some proprietary code that you don't want others to download, why not build a blink.ino update and see if that fails? You can point us at it to look at things without exposing your own stuff. OTW there's nothing the maintainers can do...

[zhfei1979]

I modified httpUpdate as following and reproduced it, you can try run the sketch with core 3.0.0, see if it happens.
The file "https://apps.95wd.com.cn/httpUpdate.ino.generic.bin" was generated with the following sketch codes too.

/**
   httpUpdate.ino

    Created on: 27.11.2015

*/

#include <Arduino.h>

#include <ESP8266WiFi.h>
#include <ESP8266WiFiMulti.h>

#include <ESP8266HTTPClient.h>
#include <ESP8266httpUpdate.h>

#ifndef APSSID
#define APSSID "GL-H6-2"
#define APPSK  "GlH62401"
#endif

ESP8266WiFiMulti WiFiMulti;

void setup() {

  Serial.begin(74880);
  Serial.setDebugOutput(true);

  Serial.println();
  Serial.println();
  Serial.println();

  for (uint8_t t = 4; t > 0; t--) {
    Serial.printf("[SETUP] WAIT %d...\n", t);
    Serial.flush();
    delay(1000);
  }

  WiFi.mode(WIFI_STA);
  WiFiMulti.addAP(APSSID, APPSK);


}

void update_started() {
  Serial.println("CALLBACK:  HTTP update process started");
}

void update_finished() {
  Serial.println("CALLBACK:  HTTP update process finished");
}

void update_progress(int cur, int total) {
  Serial.printf("CALLBACK:  HTTP update process at %d of %d bytes...\n", cur, total);
}

void update_error(int err) {
  Serial.printf("CALLBACK:  HTTP update fatal error code %d\n", err);
}


void loop() {
  // wait for WiFi connection
  if ((WiFiMulti.run() == WL_CONNECTED)) {

    BearSSL::WiFiClientSecure client;//WiFiClient client;
    client.setFingerprint("36AC62F2D4A1BC79C417E2986DFA8E85FF024DCC");

    // The line below is optional. It can be used to blink the LED on the board during flashing
    // The LED will be on during download of one buffer of data from the network. The LED will
    // be off during writing that buffer to flash
    // On a good connection the LED should flash regularly. On a bad connection the LED will be
    // on much longer than it will be off. Other pins than LED_BUILTIN may be used. The second
    // value is used to put the LED on. If the LED is on with HIGH, that value should be passed
    ESPhttpUpdate.setLedPin(LED_BUILTIN, LOW);

    // Add optional callback notifiers
    ESPhttpUpdate.onStart(update_started);
    ESPhttpUpdate.onEnd(update_finished);
    ESPhttpUpdate.onProgress(update_progress);
    ESPhttpUpdate.onError(update_error);

    t_httpUpdate_return ret = ESPhttpUpdate.update(client, "https://apps.95wd.com.cn/httpUpdate.ino.generic.bin");
    // Or:
    //t_httpUpdate_return ret = ESPhttpUpdate.update(client, "server", 80, "file.bin");

    switch (ret) {
      case HTTP_UPDATE_FAILED:
        Serial.printf("HTTP_UPDATE_FAILD Error (%d): %s\n", ESPhttpUpdate.getLastError(), ESPhttpUpdate.getLastErrorString().c_str());
        break;

      case HTTP_UPDATE_NO_UPDATES:
        Serial.println("HTTP_UPDATE_NO_UPDATES");
        break;

      case HTTP_UPDATE_OK:
        Serial.println("HTTP_UPDATE_OK");
        break;
    }
  }
}