Implement drivers for microwindows

[Acfboy]

Hello the community! Here is the work I have done for the first half of GSoC, now complete and ready for review.

Work Completed

The GDI-level Microwindows core, drivers, and mwdemo are functional on QEMU x86-64 and the simulator.

Build System Integration

• Based on the initial proof-of-concept branch by @ppisa, the Microwindows core (engine, drivers, precompiled bitmap fonts) has been successfully compiled into the NuttX apps library.
• Uses MWCONFIG_FILE="mwconfig.nuttx" to inject NuttX-specific configuration into Microwindows without touching upstream headers.
• Image resources used by mwdemo.c are compiled into binary using the convbmp tool during the build.

NuttX Hardware Drivers

Framebuffer driver (scr_nuttx.c), keyboard drivers (kbd_nuttx_event.c for event-mode and kbd_nuttx_raw.c for raw byte-stream), mouse driver (mou_nuttx_mouse.c), and touchscreen driver (mou_nuttx_ts.c) have been implemented and submitted to upstream Microwindows mainline ghaerr/microwindows#193. The graphics/microwindows/Makefile selects the appropriate drivers via ARCH=NUTTX and Kconfig-driven KEYBOARD/MOUSE variables.

Kconfig Options

Flexible Kconfig for keyboard driver: event-mode (/dev/kbd), raw byte-stream via kbd_codec (/dev/kbda), none, or custom. For mouse/touchscreen: relative mouse (/dev/mouse0), touchscreen (/dev/input0), none, or custom. Custom options allow BSP/app-level overrides without modifying the Microwindows build logic.

Demo Application

Ported mwdemo.c (a complex Windows-like demo) into a standalone NuttX example application. The demo runs successfully on both qemu-intel64:mw and the NuttX simulator (sim:mw).

I have also tested the demo with #define CONTROL 1 enabled in mwdemo.c, which represents a more complex use case.

Upstream Bug Fixes

Several issues were discovered, analysed, and most of them already fixed:

Issue	Root Cause	Fix
Animation freezing in mwdemo	MwSelect() blocked forever when a timer had already expired, because the code assumed no timer existed when GdGetNextTimeout returned false.	Patch by ppisa, correctly handle the case where timeout == 0 and return immediately.
Cursor leaving a blue ghost image	#if defined(ELKS) in mwtypes.h forced MWPIXELVALHW to unsigned char even when ELKS=0 (intended to disable it). This truncated 32-bit pixel values read from the framebuffer.	Changed to #if ELKS
System freeze when dragging a window caption	During drag, MwSelect switches to polling (select(…, timeout=0)). The USB input thread had a lower priority (50) than the application (100), so it never got CPU time to generate events.	Short-term: adjusted thread priorities in defconfigs. A permanent solution in Microwindows may be discussed later.
Mouse drift in simulator after dragging	TOUCH_POS_VALID flag was not being checked in the touchscreen driver, causing uninitialized coordinate values to be used.	Fixed in mou_nuttx.c by checking TOUCH_POS_VALID before using touch coordinates.
KEYCODE_xxx and ASCII range conflict	NuttX's KEYCODE_xxx macros (e.g., KEYCODE_PAUSE = 0x20) overlap with ASCII characters, making it impossible to distinguish special keys from normal input in the event-mode keyboard driver.	Still under discussions apache/nuttx#19527

How to Reproduce (QEMU / Simulator)

qemu-intel64:mw

make distclean
./tools/configure.sh qemu-intel64:mw
make -j$(nproc)

Then create a bootable disk following the documentation and run:

qemu-system-x86_64 -cpu host -enable-kvm -m 2G \
        -cdrom boot_mw.iso \
        -serial stdio -monitor none -no-reboot -device qemu-xhci -device usb-kbd -device usb-mouse

sim:mw

make distclean
./tools/configure.sh sim:mw
make -j$(nproc)
./nuttx

Notice

• You can set #define CONTROL 1 manually in mwdemo.c to test more complex case.
• Most of the nxstyle problems in mwdemo.c are fixed, except the names of Win32 API.
• A draft document for microwindows is at Acfboy/nuttx@f4d9316
• The Nano-X X11 supprot is still work in progress.

[ppisa]

Thanks to @Acfboy for the work and this draft request for discussion and @ghaerr for consultations and support on Microwidows side.

I am leaving for four days now, but there are some my thoughts, the NuttX Microwidows screen, mouse and keyboard drivers should be submitted with appropriate build enable options to the mainline Microwindows https://github.com/ghaerr/microwindows before the final nuttx-apps pull request.

[acassis]

@Acfboy please fix the issues on mwdemo:

[simbit18]

Hi @Acfboy , could you also add support for building with CMake?

[Acfboy]

Hi @Acfboy , could you also add support for building with CMake?

Thank you for the reminder. I will add CMake support later.

[acassis]

@Acfboy could you spend some time fixing and improving this PR? It still as Draft. It is important to get it merged and let more people to test and review it. Also please remember to include a proper nuttx/Documentation about nanox/microwindows support on NuttX

[Acfboy]

Hi @acassis , thank for the reminder. I agree I should submit the previous work as a proper PR for the community to review. However, for it to be a proper PR, I think I need to merge the existing NuttX drivers into the Microwindows mainline first. I apologize for postponing this part of the work earlier. I will submit a PR for upstream MW review soon.

[ppisa]

Yes, this what I have proposed and expect as well.

You should set something like SCREEN=NUTTX or SCREEN=NUTTX_FB in the local graphics/microwindows/Makefile and then Microwindows with merged code should add correct driver sources when you include microwindows/src/drivers/Objects.rules.

Same for the MOUSE and KEYBOARD. The question is if the mouse should be a single driver or one for relative motion mouse device and other for touchscreen.

As for the keyboard, there can be space to think how to map it the best way. The NnuttX provides its KEYCODE_xxx macros in kbd_codec.h for special keys. But only for these and rest is expected to be ASCII. Microwindows works with MWKEY and MWSCANCODE which aligns with concept of real X11 where the keyboards provide "position" of pressed key by scancode and it can be remapped to symbol even latter in application. But usually MWKEY code is used and its definitions are provided in src/include/mwtypes.h.

You provide translation in translate_keycode when keyboard is in g_kbd_mode set to KBD_MODE_EVENT but there is heuristics during open. I think that there could be two drivers, one for event type and another for raw keyboards. This could be configured through Kconfig, same would be useful for device name. So it is necessary to think a little what is the best option there.

You should try even if Nano-X server and clients can be run on NuttX. I do not see X11 graphics option as the main priority, but it would prove that even complex setup can be build and check if it is stable on NuttX.

[Acfboy]

The NnuttX provides its KEYCODE_xxx macros in kbd_codec.h for special keys. But only for these and rest is expected to be ASCII.

Hi, @ppisa . I have a question about this statement. It seems to me that the ranges of the KEYCODE_xxx macros and ASCII are conflicting, which makes it impossible for my driver to distinguish whether it's a character or a special key. In my sim:mw setup, this results in being unable to type a space.

I think that there could be two drivers, one for event type and another for raw keyboards. This could be configured through Kconfig, same would be useful for device name.

Two drivers makes the code cleaner. But specify the device path/name via Kconfig risks inconsistency with the real device. Alternatively, if Microwindows auto-selects the device based on NuttX config, then we have to keep both sides in sync — which adds maintenance overhead.

You should try even if Nano-X server and clients can be run on NuttX.

Okay, I'll try server and client of NanoX.

[ppisa]

The NnuttX provides its KEYCODE_xxx macros in kbd_codec.h for special keys. But only for these and rest is expected to be ASCII.

Hi, @ppisa . I have a question about this statement. It seems to me that the ranges of the KEYCODE_xxx macros and ASCII are conflicting, which makes it impossible for my driver to distinguish whether it's a character or a special key. In my sim:mw setup, this results in being unable to type a space.

Yes, you are right, I expected that for KEYCODE_NORMAL the ASCII value is returned in another field of the event structure, but there is only

struct keyboard_event_s
{
  uint32_t type;
  uint32_t code;
};

where type is KEYBOARD_PRESS or KEYBOARD_RELEASE. But into code sent to keyboard_event() is ASCII code for example in g_km_keymap in STM32 matrix keyboard but some other drivers call keyboard_event() even with X11 encoded KeySim (arch/sim/src/sim/posix/sim_x11eventloop.c) and some with keys translated to KEYCODE_xxx where is no space for ASCII.

So this seems to be call for priority issue.

@acassis @gregory-nutt Please, do you have some some insight what is right?

The simple fix is to push KEYCODE_FWDDEL in include/nuttx/input/kbd_codec.h to be above ASCII, for example 128 or 129. But may it be it can break some targets where it would result in some large arrays, etc. When keyboards with interpreted national symbols are used then it would worth to push special codes even further after basic unicode...

I think that there could be two drivers, one for event type and another for raw keyboards. This could be configured through Kconfig, same would be useful for device name.

Two drivers makes the code cleaner. But specify the device path/name via Kconfig risks inconsistency with the real device. Alternatively, if Microwindows auto-selects the device based on NuttX config, then we have to keep both sides in sync — which adds maintenance overhead.

Yes but NuttX is based on configuration and it is better to fail then to do lot of testing of different device names and even deciding if it is raw driver or event based. But in general it seems that there is pace for some discussion and making KBD simple on the NuttX side.

You should try even if Nano-X server and clients can be run on NuttX.

Okay, I'll try server and client of NanoX.

Thanks.

[acassis]

The NnuttX provides its KEYCODE_xxx macros in kbd_codec.h for special keys. But only for these and rest is expected to be ASCII.

Hi, @ppisa . I have a question about this statement. It seems to me that the ranges of the KEYCODE_xxx macros and ASCII are conflicting, which makes it impossible for my driver to distinguish whether it's a character or a special key. In my sim:mw setup, this results in being unable to type a space.

Yes, you are right, I expected that for KEYCODE_NORMAL the ASCII value is returned in another field of the event structure, but there is only

struct keyboard_event_s
{
  uint32_t type;
  uint32_t code;
};

where type is KEYBOARD_PRESS or KEYBOARD_RELEASE. But into code sent to keyboard_event() is ASCII code for example in g_km_keymap in STM32 matrix keyboard but some other drivers call keyboard_event() even with X11 encoded KeySim (arch/sim/src/sim/posix/sim_x11eventloop.c) and some with keys translated to KEYCODE_xxx where is no space for ASCII.

So this seems to be call for priority issue.

@acassis @gregory-nutt Please, do you have some some insight what is right?

The simple fix is to push KEYCODE_FWDDEL in include/nuttx/input/kbd_codec.h to be above ASCII, for example 128 or 129. But may it be it can break some targets where it would result in some large arrays, etc. When keyboards with interpreted national symbols are used then it would worth to push special codes even further after basic unicode...

I think that there could be two drivers, one for event type and another for raw keyboards. This could be configured through Kconfig, same would be useful for device name.

Two drivers makes the code cleaner. But specify the device path/name via Kconfig risks inconsistency with the real device. Alternatively, if Microwindows auto-selects the device based on NuttX config, then we have to keep both sides in sync — which adds maintenance overhead.

Yes but NuttX is based on configuration and it is better to fail then to do lot of testing of different device names and even deciding if it is raw driver or event based. But in general it seems that there is pace for some discussion and making KBD simple on the NuttX side.

You should try even if Nano-X server and clients can be run on NuttX.

Okay, I'll try server and client of NanoX.

Thanks.

Hi @ppisa I don't know the right approach to keyboard input keys symbols. Actually @linguini1 faced similar issue when porting Doom to NuttX. Unfortunately NuttX doesn't have a standard keyboard symbols

[linguini1]

One option suggested by @cederom was to adopt the EVDEV codec: https://en.wikipedia.org/wiki/Evdev

[ppisa]

Hi @ppisa I don't know the right approach to keyboard input keys symbols. Actually @linguini1 faced similar issue when porting Doom to NuttX. Unfortunately NuttX doesn't have a standard keyboard symbols

@linguini1 and @acassis, I propose simple fix include/nuttx/input/kbd_codec.h

-  KEYCODE_FWDDEL,
+  KEYCODE_FWDDEL = 128,     

Or even larger shift to something like 0xF800 as used on Microwindows.

This would solve problem with overlap of letters and other ASCII codes with special keys.

I can open issue for this. The question is if there is something, some other keyboard driver or application, which breaks by this shift of KEYCODE_xxx. It would be problem if somebody uses the codes as index into some array etc... 128 or 129 shift is not so big so it less expensive if somebody uses arrays for some key code transformation.

I think that that this is reasonable solution for NuttX and current model.

[linguini1]

I agree, and this was a solution I considered but ultimately did not have enough time to verify if it would break other things in the kernel. I think that is at least a good interim solution which is less invasive.

[ghaerr]

Or even larger shift to something like 0xF800 as used on Microwindows.

Looking ahead at Unicode, Microwindows moved its key code base to F800 because that belongs to the Unicode BMP Private Use Area. This won't overlap other Unicode values if/when NuttX applications move to Unicode.

It would be problem if somebody uses the codes as index into some array etc... 128 or 129 shift is not so big

Possibly good idea, but pushes off actively looking through source for problems now, and not having to change them again perhaps later. FWIW, myself knowing very little about NuttX, if a small shift is thought better for now, choosing a base of 256 at least allows space for using the upper half of various 256-byte multilingual code pages for non-US (e.g. European) accented characters, etc. OTOH, if keycodes are stored in byte arrays, then 128/129 would likely be mandated.

[xiaoxiang781216]

fix the comment

[Acfboy]

Thanks for the careful review!

[xiaoxiang781216]

ditto

[xiaoxiang781216]

remove

[xiaoxiang781216]

remove

[xiaoxiang781216]

add the dependence to Kconfig

[Acfboy]

thanks!

[xiaoxiang781216]

@Acfboy please clean up your patch before merging, @acassis don't merge this patch before I approve.

[Acfboy]

Hi @xiaoxiang781216 @acassis thanks for the thorough review! I've addressed all your feedback and fixed the non-standard commit messages flagged by checkpatch.

[xiaoxiang781216]

Hi @xiaoxiang781216 @acassis thanks for the thorough review! I've addressed all your feedback and fixed the non-standard commit messages flagged by checkpatch.

please merge your temp change.