Merge 782a35bf9e into 3278173ef8

The implementation is much more straightforward.

It should also solve a JDK incompatiblity:

  java.lang.NoSuchMethodError: java.nio.ByteBuffer.flip()Ljava/nio/ByteBuffer;
  at processing.app.Serial.serialEvent(Serial.java:185)
  at jssc.SerialPort$LinuxEventThread.run(SerialPort.java:1299)

See #8903

README.md

@@ -1,97 +1,59 @@
 <p align="center">
-	<img src="http://content.arduino.cc/brand/arduino-color.svg" width="50%" />
+    <img src="http://content.arduino.cc/brand/arduino-color.svg" width="50%" />
 </p>
 
-Arduino is an open-source physical computing platform based on a simple I/O
-board and a development environment that implements the Processing/Wiring
-language. Arduino can be used to develop stand-alone interactive objects or
-can be connected to software on your computer (e.g. Flash, Processing and MaxMSP).
-The boards can be assembled by hand or purchased preassembled; the open-source
-IDE can be downloaded for free at [https://arduino.cc](https://www.arduino.cc/en/Main/Software)
+**Important Notice**: This repository contains the legacy Arduino IDE 1.x, which is no longer in active development. For the latest features and updates, please visit the [Arduino IDE 2.x](https://github.com/arduino/arduino-ide) repository. If you encounter issues related to the newer IDE, please report them there.
+
+Arduino is an open-source physical computing platform based on a simple I/O board and a development environment that implements the Processing/Wiring language. Arduino can be used to develop stand-alone interactive objects or can be connected to software on your computer (e.g. Flash, Processing and MaxMSP). The boards can be assembled by hand or purchased preassembled; the open-source IDE can be downloaded for free at [https://arduino.cc](https://www.arduino.cc/en/Main/Software).
 
 ![Github](https://img.shields.io/github/v/release/arduino/Arduino)
 
 ## More info at
 
--  [Our website](https://www.arduino.cc/)
-
--  [The forums](https://forum.arduino.cc/)
-
--  Follow us on [Twitter](https://twitter.com/arduino)
--  And like us at [Facebook](https://www.facebook.com/official.arduino)
+- [Our website](https://www.arduino.cc/)
+- [The forums](https://forum.arduino.cc/)
+- Follow us on [Twitter](https://twitter.com/arduino)
+- And like us at [Facebook](https://www.facebook.com/official.arduino)
 
 ## Bug reports and technical discussions
 
--  To report a *bug* in the software or to request *a simple enhancement* go to [Github Issues](https://github.com/arduino/Arduino/issues)
-
--  More complex requests and technical discussion should go on the [Arduino Developers
-mailing list](https://groups.google.com/a/arduino.cc/forum/#!forum/developers)
-
--  If you're interested in modifying or extending the Arduino software, we strongly
-suggest discussing your ideas on the
-[Developers mailing list](https://groups.google.com/a/arduino.cc/forum/#!forum/developers)
- *before* starting to work on them.
-That way you can coordinate with the Arduino Team and others,
-giving your work a higher chance of being integrated into the official release
+- To report a *bug* in the software or to request *a simple enhancement*, go to [Github Issues](https://github.com/arduino/Arduino/issues).
+- More complex requests and technical discussions should go on the [Arduino Developers mailing list](https://groups.google.com/a/arduino.cc/forum/#!forum/developers).
+- If you're interested in modifying or extending the Arduino software, we strongly suggest discussing your ideas on the [Developers mailing list](https://groups.google.com/a/arduino.cc/forum/#!forum/developers) *before* starting to work on them. That way you can coordinate with the Arduino Team and others, giving your work a higher chance of being integrated into the official release.
 
 ### Security
 
-If you think you found a vulnerability or other security-related bug in this project, please read our
-[security policy](https://github.com/arduino/Arduino/security/policy) and report the bug to our Security Team 🛡️
-Thank you!
+If you think you found a vulnerability or other security-related bug in this project, please read our [security policy](https://github.com/arduino/Arduino/security/policy) and report the bug to our Security Team 🛡️. Thank you!
 
 e-mail contact: security@arduino.cc
 
 ## Installation
 
-Detailed instructions for installation in popular operating systems can be found at:
+Detailed instructions for installation on popular operating systems can be found at:
 
--  [Linux](https://www.arduino.cc/en/Guide/Linux) (see also the [Arduino playground](https://playground.arduino.cc/Learning/Linux))
--  [macOS](https://www.arduino.cc/en/Guide/macOS)
--  [Windows](https://www.arduino.cc/en/Guide/Windows)
+- [Linux](https://www.arduino.cc/en/Guide/Linux) (see also the [Arduino playground](https://playground.arduino.cc/Learning/Linux))
+- [macOS](https://www.arduino.cc/en/Guide/macOS)
+- [Windows](https://www.arduino.cc/en/Guide/Windows)
 
 ## Contents of this repository
 
-This repository contains just the code for the Arduino IDE itself.
-Originally, it also contained the AVR and SAM Arduino core and libraries
-(i.e.  the code that is compiled as part of a sketch and runs on the
-actual Arduino device), but those have been moved into their own
-repositories.  They are still automatically downloaded as part of the
-build process and included in built releases, though.
+This repository contains just the code for the Arduino IDE itself. Originally, it also contained the AVR and SAM Arduino core and libraries (i.e. the code that is compiled as part of a sketch and runs on the actual Arduino device), but those have been moved into their own repositories. They are still automatically downloaded as part of the build process and included in built releases, though.
 
 The repositories for these extra parts can be found here:
--   Non-core specific Libraries are listed under: <https://github.com/arduino-libraries/>
-    (and also a few other places, see `build/build.xml`).
-
--   The AVR core can be found at: <https://github.com/arduino/ArduinoCore-avr>
-
--   Other cores are not included by default but installed through the
-    board manager. Their repositories can also be found under
-    <https://github.com/arduino/>.
+- Non-core specific Libraries are listed under: [Arduino Libraries](https://github.com/arduino-libraries/) (and also a few other places, see `build/build.xml`).
+- The AVR core can be found at: [ArduinoCore-avr](https://github.com/arduino/ArduinoCore-avr).
+- Other cores are not included by default but can be installed through the board manager. Their repositories can also be found under [Arduino GitHub organization](https://github.com/arduino/).
 
 ## Building and testing
 
-Instructions for building the IDE and running unit tests can be found on
-the wiki:
--   <https://github.com/arduino/Arduino/wiki/Building-Arduino>
--   <https://github.com/arduino/Arduino/wiki/Testing-Arduino>
+Instructions for building the IDE and running unit tests can be found on the wiki:
+- [Building Arduino](https://github.com/arduino/Arduino/wiki/Building-Arduino)
+- [Testing Arduino](https://github.com/arduino/Arduino/wiki/Testing-Arduino)
 
 ## Credits
 
-Arduino is an open source project, supported by many.
+Arduino is an open-source project, supported by many. The Arduino team is composed of Massimo Banzi, David Cuartielles, Tom Igoe, and David A. Mellis.
 
-The Arduino team is composed of Massimo Banzi, David Cuartielles, Tom Igoe
-and David A. Mellis.
-
-Arduino uses
-[GNU avr-gcc toolchain](https://gcc.gnu.org/wiki/avr-gcc),
-[GCC ARM Embedded toolchain](https://launchpad.net/gcc-arm-embedded),
-[avr-libc](https://www.nongnu.org/avr-libc/),
-[avrdude](https://www.nongnu.org/avrdude/),
-[bossac](http://www.shumatech.com/web/products/bossa),
-[openOCD](http://openocd.org/)
-and code from [Processing](https://www.processing.org)
-and [Wiring](http://wiring.org.co).
-
-Icon and about image designed by [ToDo](https://www.todo.to.it/)
+Arduino uses [GNU avr-gcc toolchain](https://gcc.gnu.org/wiki/avr-gcc), [GCC ARM Embedded toolchain](https://launchpad.net/gcc-arm-embedded), [avr-libc](https://www.nongnu.org/avr-libc/), [avrdude](https://www.nongnu.org/avrdude/), [bossac](http://www.shumatech.com/web/products/bossa), [openOCD](http://openocd.org/), and code from [Processing](https://www.processing.org) and [Wiring](http://wiring.org.co).
 
+Icon and about image designed by [ToDo](https://www.todo.to.it/).

app/test/processing/app/SerialTest.java

@@ -29,6 +29,7 @@
 
 package processing.app;
 
+import static org.junit.Assert.assertArrayEquals;
 import static org.junit.Assert.assertEquals;
 
 import org.junit.Test;
@@ -47,6 +48,16 @@ public class SerialTest {
     String output = "";
   }
 
+  @Test
+  public void testSerialUTF8DecoderWithInvalidChars() throws Exception {
+    NullSerial s = new NullSerial();
+    byte[] testdata = new byte[] { '>', (byte) 0xC3, (byte) 0x28, '<' };
+    byte[] expected = new byte[] { '>', (byte) 0xEF, (byte) 0xBF, (byte) 0xBD, (byte) 0x28, '<' };
+    s.processSerialEvent(testdata);
+    byte[] res = s.output.getBytes("UTF-8");
+    assertArrayEquals(expected, res);
+  }
+
   @Test
   public void testSerialUTF8Decoder() throws Exception {
     NullSerial s = new NullSerial();

arduino-core/src/processing/app/Serial.java

@@ -22,23 +22,22 @@
 
 package processing.app;
 
-import jssc.SerialPort;
-import jssc.SerialPortEvent;
-import jssc.SerialPortEventListener;
-import jssc.SerialPortException;
+import static processing.app.I18n.format;
+import static processing.app.I18n.tr;
 
 import java.io.IOException;
-import java.nio.ByteBuffer;
-import java.nio.CharBuffer;
+import java.io.InputStreamReader;
+import java.io.PipedInputStream;
+import java.io.PipedOutputStream;
 import java.nio.charset.Charset;
-import java.nio.charset.CharsetDecoder;
-import java.nio.charset.CodingErrorAction;
 import java.nio.charset.StandardCharsets;
 import java.util.Arrays;
 import java.util.List;
 
-import static processing.app.I18n.format;
-import static processing.app.I18n.tr;
+import jssc.SerialPort;
+import jssc.SerialPortEvent;
+import jssc.SerialPortEventListener;
+import jssc.SerialPortException;
 
 public class Serial implements SerialPortEventListener {
 
@@ -53,11 +52,9 @@ public class Serial implements SerialPortEventListener {
 
   private SerialPort port;
 
-  private CharsetDecoder bytesToStrings;
-  private static final int IN_BUFFER_CAPACITY = 128;
-  private static final int OUT_BUFFER_CAPACITY = 128;
-  private ByteBuffer inFromSerial = ByteBuffer.allocate(IN_BUFFER_CAPACITY);
-  private CharBuffer outToMessage = CharBuffer.allocate(OUT_BUFFER_CAPACITY);
+  private PipedOutputStream decoderInRaw;
+  private InputStreamReader decoderOutputUTF8;
+  private final int DECODER_BUFF_SIZE = 16384;
 
   public Serial() throws SerialException {
     this(PreferencesData.get("serial.port"),
@@ -189,42 +186,18 @@ public class Serial implements SerialPortEventListener {
 
   public void processSerialEvent(byte[] buf) {
     int next = 0;
-    // This uses a CharsetDecoder to convert from bytes to UTF-8 in
-    // a streaming fashion (i.e. where characters might be split
-    // over multiple reads). This needs the data to be in a
-    // ByteBuffer (inFromSerial, which we also use to store leftover
-    // incomplete characters for the nexst run) and produces a
-    // CharBuffer (outToMessage), which we then convert to char[] to
-    // pass onwards.
-    // Note that these buffers switch from input to output mode
-    // using flip/compact/clear
-    while (next < buf.length || inFromSerial.position() > 0) {
-      do {
-        // This might be 0 when all data was already read from buf
-        // (but then there will be data in inFromSerial left to
-        // decode).
-        int copyNow = Math.min(buf.length - next, inFromSerial.remaining());
-        inFromSerial.put(buf, next, copyNow);
-        next += copyNow;
-
-        inFromSerial.flip();
-        bytesToStrings.decode(inFromSerial, outToMessage, false);
-        inFromSerial.compact();
-
-        // When there are multi-byte characters, outToMessage might
-        // still have room, so add more bytes if we have any.
-      } while (next < buf.length && outToMessage.hasRemaining());
-
-      // If no output was produced, the input only contained
-      // incomplete characters, so we're done processing
-      if (outToMessage.position() == 0)
-        break;
-
-      outToMessage.flip();
-      char[] chars = new char[outToMessage.remaining()];
-      outToMessage.get(chars);
-      message(chars, chars.length);
-      outToMessage.clear();
+    int max = buf.length;
+    char chars[] = new char[DECODER_BUFF_SIZE];
+    try {
+      while (next < max) {
+        int w = Integer.min(max - next, chars.length);
+        decoderInRaw.write(buf, next, w);
+        next += w;
+        int n = decoderOutputUTF8.read(chars);
+        message(chars, n);
+      }
+    } catch (IOException e) {
+      e.printStackTrace();
     }
   }
 
@@ -295,10 +268,14 @@ public class Serial implements SerialPortEventListener {
    * before they are handed as Strings to {@Link #message(char[], int)}.
    */
   public synchronized void resetDecoding(Charset charset) {
-    bytesToStrings = charset.newDecoder()
-                      .onMalformedInput(CodingErrorAction.REPLACE)
-                      .onUnmappableCharacter(CodingErrorAction.REPLACE)
-                      .replaceWith("\u2e2e");
+    try {
+      decoderInRaw = new PipedOutputStream();
+      // add 16 extra bytes to make room for incomplete UTF-8 chars
+      decoderOutputUTF8 = new InputStreamReader(new PipedInputStream(decoderInRaw, DECODER_BUFF_SIZE + 16), charset);
+    } catch (IOException e) {
+      // Should never happen...
+      e.printStackTrace();
+    }
   }
 
   static public List<String> list() {