Arduino IDE sometimes freezes when uploading the code

fanmingyu212

Hello,

I am using Quarto with Arduino IDE (v2.2.1 on Ubuntu 22.04) with QNimble board v1.9.0 installed. Sometimes when I upload the code, Arduino IDE freezes on "Uploading". When this happens, the Quarto serial ports changes (the two ports cycle between ttyACM0,1,2). After a few cycles of closing the IDE, selecting the serial port, and uploading, it usually works and uploads the code successfully. Sometimes I need to restart the Quarto and go into the bootloader to make it work. My observations are similar to https://forum.qnimble.com/d/4-hello-world-on-linux.

I don't have any other program using the serial port other than the Arduino IDE. I observed this problem no matter if the serial monitor is closed or opened. I tried multiple sketches, including example sketches (e.g. DeviceInfo from quarto-examples repo), and I have observed the problem with all of them.

Is there some way I can make the uploading process more robust?

Ben

I'm sorry you are having trouble with your Quarto. I believe most of the issues in the older post have been addressed now that the Arduino IDE filters out the ports correctly. The Arduino IDE should always bind to the programming port, even if it changes from ttyACM0, and ttyACM1, etc. That is assuming you have the Arduino IDE connect to the 'qnimble port' (and not the 'Serial port':

This is a long shot, but in your setup, do you have the Quarto connected to your computer via a long USB cable (>6 ft) or a USB hub? If so, can you try connecting the Quarto directly to the computer with a short cable and see if that makes a difference.

That said, I've recreated your environment (Ubuntu 22.04, Arduide IDE 2.2.1, board's v1.9.0) and I'm seeing some intermittent issues with flashing the device, so I'm going to investigate more on my end and hopefully I can find (and fix) the underlying issue.

Thank you for your patience and if you find any behavior that makes the flashing more or less reliable please let me know to help track down what's going on.

fanmingyu212

Hi Ben,

Thank you for the response. I do have the IDE connect to the qnimble port as shown in the picture you attached. However, whenever it stucks in the upload step, the port name changes and the wrong port is selected, so I would need to select the qnimble port again.

I do have a long USB cable and a USB hub between the Quarto and the computer. Currently in our experiment setup (we are using it in the context of an atomic physics lab), it is difficult for us to place the Quarto where a short USB cable can be used. I may test using a raspberry pi near the Quarto to control it and see if it helps.

Thank you for the help!

Ben

LT;DR: Please try the latest beta 1.9.2beta1 of the board support package and let me know if that helps or not.

I don't fully understand it, but the way Linux assigns ports name (ttyACM1, vs ttyACM2) seems fickle. On some computers, with some hubs, etc, when the Quarto reboots to enter programming mode the port names changes. The common way to address this in Linux is to not care and instead use udev to create a symbolic link from ttyACMX to, say ,"MyDevice" and then access the "MyDevice" and let udev handle making sure that points to whatever the current device is (ttyACM0, or ttyACM1). Unfortunately, this doesn't play that nicely with how the Arduino IDE filters and selects for ports. (And there are some additional challenges with udev and accessing bInterfaceNumber to get the right Quarto port).

What I've found is that Linux botches the reboot and renaming when a program queries for the port too soon. So if you reboot the device, twiddle your thumbs and then reconnect to it, the naming seems to consistently stay preserved. If you reboot it and try to connect to the port before it comes back, then I think that keeps the port open longer in the OS and when the rebooted Quarto device shows up, it gets assigned a new port as the old one it still hanging around. Because different hardware and OS's take different amounts of time to see a new device the programming software tries to connect soon after rebooting the Quarto and runs in a loop until that connection is successful. This seems to sometimes trigger the problem on Linux, depending on the timing of everything. I've made two changes to the programmer. One is to simply wait longer for the Quarto to reboot (Linux only). And the second, also Linux only, is to look for the existence of the serial port without trying to open it as this seems to trigger the issue. That's in qbossa 1.9.7 and should be installed with the 1.9.2beta1 software. Let me know how it goes.

Also, to get access to the beta software, set the Additional boards manager url to

 https://qnimble.com/package_qnimble_beta_index.json

fanmingyu212

I tried the latest beta 1.9.2beta1 and unfortunately it did not help in my case.

I observed the same problem as before: Quarto port changes when I press the upload button, and then uploading freezes. I had to close the IDE, select the port, and upload for multiple times before it succeeds. On average it takes around ~5 repeats before the code is uploaded (similar to the experience with the previous software version 1.9.0). I should note that I am still using the long USB cable + hub to connect to it currently though, so possibly the long USB cable is also part of the problem.

Ben

I'm sorry that the update did not fix the behavior. Unfortunately, I think that puts us in the situation where I'm unable to replicate the issue you are seeing so I'll need more help from you to isolate / troubleshoot the issue. Please feel free to use the contact link to switch to direction communication if you prefer.

As a first step, can you try the following command:

~/username/.arduino15/packages/qnimble/tools/qbossa/1.9.7/qbossac  -p /dev/ttyACM0 -a -i -R

the exact path may be bit different, but if you turn on verbose output for upload in the Arduino IDE preferences, when you click upload, it should show the full command to uploade a sketch, including the path to the uploader qbossac.

That command should cause the Quarto to reboot into the bootloader, display some basic info and then boot into your application. The output should be something like

$  ~username/.arduino15/packages/qnimble/tools/qbossa/1.9.7/qbossac  -p /dev/ttyACM0 -a -i -R
Restablished Connection on loop=2
Device       : qNimble BOSSAv1
Version      : v0.5.0-dev [Arduino:XYZ] Jan 29 2023
Address      : 0x300000
Pages        : 32768
Page Size    : 256 bytes
Total Size   : 8192KB
Planes       : 1
Lock Regions : 32
Locked       : 16
Security     : false
Boot Flash   : false
BOD          : false
BOR          : false

When you run that the Quarto should end up in the same state as it started with the same port names. Does that work consistently, or does that trigger the port renaming some of the time?

I'm guessing that this will cause the problem. If it doesn't, can you try again without the "-i" flag to qbossac and see if that makes a difference?

Assuming the qbossac command causes the problem, can you try the following python program:

import serial
dev = '/dev/ttyACM0'

ser=serial.Serial(dev,1200,timeout=1)
del ser

This should also cause the Quarto to reboot and go to the bootloader for a few seconds and then boot back into your application. If this works consistently, then I think the issue is with the timing of when the qbossac program reconnects to Quarto when it gets into the bootloader (and Linux having port naming issues when trying to open a port that is in the process of disconnecting and getting reconnected).

Let me know the results either here or via contact and we'll go from there. Sorry for your issues.

fanmingyu212

Thank you for the detailed instructions. The qbossac -p /dev/ttyACM0 -a -i -R command does trigger the port renaming most of the times, and sometimes it gives the output below. The loop number in the first line is usually 5, but I have seen loop=0 and loop=6 occasionally too. When it changes port, I change /dev/ttyACM0 to /dev/ttyACM1 so it will trigger the port change again to change it back.

$ ~/.arduino15/packages/qnimble/tools/qbossa/1.9.7/qbossac  -p /dev/ttyACM0 -a -i -R
Restablished Connection on loop=5
Device       : qNimble BOSSAv1
Version      : v0.5.1 [Arduino:XYZ] May  5 2023
Address      : 0x300000
Pages        : 32768
Page Size    : 256 bytes
Total Size   : 8192KB
Planes       : 1
Lock Regions : 32
Locked       : 16
Security     : false
Boot Flash   : false
BOD          : false
BOR          : false

The python program you attached can work consistently if I wait for a few seconds from deleting the ser object to connecting again. If I put the serial connection and deleting code in a loop, with 1 s sleep between each loop, then I can see 1 failure every 4 successes very consistently. The code I tested with is attached below:

import serial
dev = '/dev/ttyACM0'

counters = []
for kk in range(50):
     try:
         ser = serial.Serial(dev, 1200, timeout=1)
         del ser
         counters.append(1)
     except:
         counters.append(0)
     time.sleep(1)
print(counters)

I get an output of [1, 1, 1, 1, 0, 1, 1, 1, 1, 0, 1, 1, 1, 1, 0, 1, 1, 1, 1, 0, 1, 1, 1, 1, 0, 1, 1, 1, 1, 0, 1, 1, 1, 1, 0, 1, 1, 1, 1, 0, 1, 1, 1, 1, 0, 1, 1, 1, 1, 0]. I never see the devices renamed when I run the code while running ls in /dev. When it fails, ls simply does not show any /dev/ttyACM* device. If I change the sleep time to 5 s, it succeeds consistently.

Ben

Thanks, I think we are getting closer. Can you run the following python program:

import serial, time
dev = '/dev/ttyACM0'

ser = serial.Serial(dev, 1200, timeout=1) ## connect as 1200 buad to reboot into bootloader
del ser
start = time.time()
time.sleep(2.5)  ## vary this to see what the minimum time needed is

while(True):
    try:
        ser = serial.Serial(dev, 115200, timeout=1) ##connect to bootloader
        print('{:.2f}: Connected to bootloader'.format(time.time()-start))
        ser.write(b'V#')   ## query version
        print(ser.readline()) ## print version
        ser.write(b'l#') ## boot application
        break
    except Exception as e:
        print('{:.2f}: Got Error: {}'.format(time.time()-start,e))
        time.sleep(.01)
        continue
del ser

Hopefully, this will run fine and consistently. It is basically what the qbossac program does except with a big sleep in it. If you set it too long (>3 or so) then the Quarto boots into the application before the python script connects to the bootloader and you won't get the bootloader version printed out. You should see an output like:

b'v0.5.0-dev [Arduino:XYZ] Jan 29 2023\n'
2.50: Connected to bootloader

Assuming this runs consistently without port renaming, can you try decreasing the time.sleep and see when it fails?

Ideally, this sleep can be set very low (_100ms, or even zero) and the serial.Serial in theloop will fail a bunch since the port won't be ready and then it'll connect. That's how qbossac works. But I think that on your system (Linux in general maybe?) if that first connection attempt happens to quickly then the port number changes. qbossac has a delay, but I suspect it isn't large enough for your system and maybe a usb hub increases how long that delay needs to be. If you find a time where it sometimes works and sometimes trigger a port renaming, it would be interesting to know if you can correlate that behavior with the time it takes to connect (although it may just fail in those cases).

fanmingyu212

Thank you for looking into this problem. I ran the code you provided with different sleep times and found the following:
When the sleep time is greater than 0.91 s on my computer, it connects with no error. For example, with 0.92 s sleep time, I get the following message consistently.

0.92: Connected to bootloader
b'v0.5.1 [Arduino:XYZ] May  5 2023\n'

When the sleep time is between 0.45 s and 0.85 s, it will return the following message (0.85 s used as an example). It will try a few times but eventually it could connect. The time that it is connected ranges from 0.87 to 0.91 s on my computer.

0.85: Got Error: [Errno 2] could not open port /dev/ttyACM0: [Errno 2] No such file or directory: '/dev/ttyACM0'
0.86: Got Error: [Errno 2] could not open port /dev/ttyACM0: [Errno 2] No such file or directory: '/dev/ttyACM0'
0.87: Got Error: [Errno 13] could not open port /dev/ttyACM0: [Errno 13] Permission denied: '/dev/ttyACM0'
0.88: Got Error: [Errno 13] could not open port /dev/ttyACM0: [Errno 13] Permission denied: '/dev/ttyACM0'
0.89: Connected to bootloader
b'v0.5.1 [Arduino:XYZ] May  5 2023\n'

When the sleep time is less than 0.4 s, it will return the following messages (0.4 s used as an example), and the serial ports will change. The "No such file or directory" error message repeats forever.

0.40: Connected to bootloader
0.44: Got Error: device reports readiness to read but returned no data (device disconnected or multiple access on port?)
0.45: Got Error: [Errno 2] could not open port /dev/ttyACM0: [Errno 2] No such file or directory: '/dev/ttyACM0'
0.46: Got Error: [Errno 2] could not open port /dev/ttyACM0: [Errno 2] No such file or directory: '/dev/ttyACM0'
0.47: Got Error: [Errno 2] could not open port /dev/ttyACM0: [Errno 2] No such file or directory: '/dev/ttyACM0'
0.48: Got Error: [Errno 2] could not open port /dev/ttyACM0: [Errno 2] No such file or directory: '/dev/ttyACM0'
0.49: Got Error: [Errno 2] could not open port /dev/ttyACM0: [Errno 2] No such file or directory: '/dev/ttyACM0'
...

I am not sure where the ~0.44 s number comes from. However, if it waits for longer than this time before connects to the bootloader, it seems to work reliably.

Edit: I was not able to correlate the number with the connection time. If I run the code below to measure the connection time:

t0 = time.time()
ser = serial.Serial(dev, 1200, timeout=1)
t1 = time.time()
del ser
t2 = time.time()
time.sleep(1)
t3 = time.time()
ser = serial.Serial(dev, 115200, timeout=1)
ser.write(b'V#')   ## query version
print(ser.readline()) ## print version
ser.write(b'l#') ## boot application
t4 = time.time()
del ser
t5 = time.time()
print(t1 - t0, t2 - t1, t4 - t3, t5 - t4)

I get similar output as the following. The times are consistent to ₁ ms.

b'v0.5.1 [Arduino:XYZ] May  5 2023\n'
0.0015709400177001953 0.0015993118286132812 0.0008213520050048828 0.12640786170959473

Ben

Thanks for the update, this is very helpful. So what's going on is that first connect (with 1200 baud) reboots the Quarto. Ideally the behavior would look like this:

Reboot -> No USB -> USB (on same port) in Bootloader

Any connection attempts during the 'No USB' time will fail, but once the USB comes up it'll connect. This is what is happening when you have the sleep between 0.45s and 0.81s. The first connect attempt fails because the USB isn't up yet but then it succeeds. However, on your setup I think you actually have:

Reboot (Time: 0) -> Danger Zone USB Ghost (Time: 0 - 0.45s) -> No USB (Time: 0.45s - 0.8s) -> USB in Bootloader (Time: 0.8s onward)

If this is right, then the trick is to make sure the program (python script or qbossac) doesn't try to connect to the device in this 'danger zone' where I believe Linux says the USB device is connected, but when you try to connect it fails but doesn't fail fast enough to have that port open when the Quarto USB reconnects.

I'm going to see if I can find a way to detect the danger zone without triggering the port naming issue as that would be ideal. I'll also see if I can get some measure of how much this timing varies so I can try to put a delay long enough to be reliable for all systems, but not unnecessary slow.

In the mean time if you want a temporary fix, you can clone the qbossa program (https://github.com/qnimble/bossa) and edit line 398 of src/bossac.cpp to increase the delay. (Its currently 200us + 200us for Linux, so just shy of the 0.45s you measured. Maybe try 400,000us for 600ms total. Then compile it with 'make bossac'. That will make a new binary (qbossac) that hopefully will work well for you. If it does, copy it to ~/Arduino15/packages/qnimble/blah/blah and overwrite the 'proper' version of qbossac. Then you can use the Arduino IDE and upload will use this modified program for the uploads. Anytime you update the board support package (1.9.1->1.9.2 for example) you'll lose your modified binary and have to copy it over again. So this isn't a real fix, but hopefully it can work until I can make a release with a better solution.

If anyone else is watching this thread and has a Linux system and wants to try the above python script and report their timing, that would be great to get more data on the timing for different systems.

Ben

Please give qbossac 1.9.8 a try. You can download it at 1.9.8 release.

Update: I think I have a solution. Instead of disconnecting from the Quarto after telling it to reboot, we keep the connection open and wait for the OS to report an error on the connection. This forces the new connection to the Quarto to happen after the OS realizes the USB has been disconnected and delays the new connection to the Quarto by an amount set by the OS seeing the USB update instead of a fixed timeout that varies a lot between computers and OS's etc.

Let me know how this works and if works well (and internal testing keeps looking good), I'll include this in an updated board package.

fanmingyu212

I tried qbossac 1.9.8 and I can get consistent uploads with no problem. Thank you for spending the time and effort to investigate and fix the problem!

Ben

Of course. Thanks for your help tracking down the problem.

I've published 1.9.3beta1 which uses the 1.9.8 version of qbossa so you (or anyone) should be able to upgrade your Arduino IDE to that and get better uploading in Linux. I'll include this update in the next non-beta release.