About this talk
This talk covers the integration of the Skia graphics library into the Standard Widget Toolkit (SWT) to enhance its capabilities. Dennis Unamar, a software developer at SAP, discusses the limitations of SWT, particularly its lack of GPU support, smartphone compatibility, and server-side rendering. He explains the choice of Skia as it is a reliable 2D graphics library with broad platform support including iOS, Android, and web browsers. The speaker demonstrates using Skia through Java bindings to create a seamless drawing experience while minimizing changes to existing code. He highlights the benefits in performance and ease of use, along with some current limitations like font handling and API compatibility. The session concludes with insights on the project's future integration plans within the SWT framework.
Full transcript
Can I start? Um, welcome to my presentation of the Skia canvas integr integration into SVT. Um, my name is Dennis Unamar and I'm software developer at SAP and I'm also one of the contributors uh to eclipse and >> [snorts] >> Okay. Now um I'm Dennis Unamar. I'm a software developer at SAP and also I'm also one of the Eclipse contributors. Um we have the intention to modernize
Eclipse and by that of course we also want to uh modernize SVT. Um actually when you look 20 years to the past um there was a um quite a big competition between swing and between SVT and the concepts of both were completely different. Um actually swing used its own customdrawn widgets and compared to that SVT relied on the native widget toolkits and just create bindings. the difference
uh in the meantime currently actually the drawbacks of this concept of SVT became quite clear. The further development is quite difficult um and actually by this integration now I want to make the situ I want to improve the situation of that. So what are some classical pain points in SVT? Um we do not have a real GPU support. Of course, we have the GL canvas and with
the GL canvas, you can actually use OpenGL, but nonetheless, you can't directly use the classical SVT API. And of course, this is some drawback. You don't have a switch where you can say, okay, no, I want to use OpenGL with the classical G um um GC drawings. Um also, you do not have a um smartphone support there. Actually, 20 years ago, it wasn't possible. The SVT is
a classical desktop widget toolkit and by that it just was never invented by that. And the last point is you do not have a serverside rendering. By them it mean I mean it's possible to just write your SVT application and just use it in the browser in the same way as you can use it on the desktop. Today actually you would expect it from a widget toolkit
but nonetheless with classical SVT it's not possible. In order to find a solution for this problems, um, Haylar who also has a talk later, um, initiated the initiative 31 and by that we came to a solution. We built a widget toolkit which completely relies on Skia. The problem of that solution is that is that the effort is quite high to create a complete widget toolkit and in
the meantime you don't get anything any real benefit for the existing SVT implementation. So as a result I've asked myself the question can we do something for SVT now in order to integrate ski directly into the widget toolkit without having to wait that a widget toolkit might be developed somewhere until that we have nothing. So I tried to produ uh I've initiated the side project by trying
to integration. Now um so what's the reason for using skier in SVT? For that let's first go shortly to the question what Skia is. It's a 2D graphics library uh written in C++. And about the rest here on the screen, I would rather say it supports basically everything that you can imagine, which actually means for SVT, we've got a huge benefit if because especially since we would
have iOS, Android, and web browser support, which means we could easily then use our drawings also in the browser and on mobiles. Um, I think we also do not have to care whether the support for Skia ends sometimes because it's probably the most established 2D graphics library in the world. And so I would guess it's very very reliable to use. Also, the license is fine for us.
We do not have to worry about any licensing problem with BSD3. We can use it directly. So the next actual question is how can we actually use Skia in a Java envir environment for SVT for that I've used the ski jar bindings um they are these are bindings written in in Java and they provide a ski support for it um in my opinion they are really comfortable
to use and that's the reason why I've actually used it of course there are also other um alternatives like the skio bindings or you could also So of course create your own C API bindings. FFM API would also be an alternative. Um nonetheless all of these would increase the effort but on the one hand side but on the other hand side I also have to say ski
jar is still public alpha. That means from our side it might be a problem to use them on the longer run. Nonetheless for the moment at the current state of this development they are quite fine to use. Um also about the architectures some architectures are for ske are not yet fully supported. Um I also have to say about that at the current state we have we do
not have to worry about this. Um this is the side. So what's on the existing SVT side when we compare the currently existing code that must be maintained in order to have 2D drawings we say see that it's quite complicated to um actually even the maintenance of the existing 2D graphics library inside SVT is quite complex and there we also have a good reason first of all
to extend the existing libraries especially for having one library on all three operating systems but also on the long run maybe we can reduce the maintenance. It's actually not my intention in in my opinion it would be the best idea to have all of them in there and at the end say the best wins and then we can continue using it. Um I've checked out the code
of the existing implementations as you can see we've got GDIG plus on Windows quads on Mac and Cairo on Linux. Um basically they use the CPU some calculations will be moved to the GPU but nonetheless they still are CPU reliant. Um they use the classical C API and I have to say the comp they are rather complex to use and compared to that of course with Skia
and the ski job bindings it would be much easier to maintain. Um also because of the processing units there we could switch between GPU and CPU um that what you can here see here with MAC actually means that the Mac implementation is not yet supported. Nonetheless it would be also possible to just include Mac. Um one of the main points about all restrictions that I'm telling you
is they are there because of the state of development not because it would be that difficult to include them. It's just more uh I'm just more restricted probably in money ways than I have to be also in order to improve the development speed. so that's the comparison. So what's the expectation of the skier usage in SVT? Um I think it's quite clear none of you people is
here because he thinks that he wants to write now new SVT applications. Um, SVT has tons of legacy code and of course we also want to use them in the future and this brings up the intention that in order to use modern technologies we want to minimize the code modifications which are necessary to use Skia. And by that at the end of the of of of the
discussion we came to the idea that the easiest way to use it is to just activate it with a style variable. That means you can just set in your kind constructor SWTI and by that the skier drawing engine will be used and no longer the classical drawing engines. So that means the switch to the new Skia drawing engine is quite easy and by that you can use
your paint listeners in exactly the same way like before everything stays the same. You just have to activate ski in this way. Um of course this is very simple to do. Nonetheless, the first question that must be asked at this point at when we look at the API is this in general possible for all situations for all drawing methods that exist in in in SVT and about
that I have to say when you take a closer look at the API I have to say no it's not that easy in general you can't but as that what I've done till now for many many parts it is quite simple and for the positions where At the moment you can find quite easy workarounds to make a quite huge part possible altogether and for the rest further
implementations are actually necessary. So let's check out what works and where um where still problems are. And by that I'm starting with the graphics example. this is actually a classical SVT example. And as you can see, okay, first of all, um I made a very small modification to this example. And the modification is the following up here. We've got the button which is the gear button. Now
the only thing that I've modified is that this gear button now creates the can with this with this specific style variable. That means now actually skier is running and when I now disable it the classical canvas is running and as you can see there is not much of a difference which is visible. So let's switch to some examples which tell us more. Um oh yeah now you
can see it. This is actually a snippet for the the alpha blending where you can see the images they move in and out and so on and also for some text. Um the real difference is here when I disable skier and actually that's one of the points where it's good that my laptop has not that good performance because you can e directly see now the difference in
the Windows GDI performance it's much much slower and this is actually one of the points that really shows some difference in it. Um, also another issue is uh because of the double buffering. Um, usually in order to make these animations, it's necessary in the constructor to set the SVT double buffering style. This is no longer necessary. Um, by that I mean this is the animation style with
classical SVT and now with Skia actually it's not even possible to not use double buffering at this point. So you do not no longer have to worry about this mode. Um also other features work quite well. This is for example transformations uh in Skia and some alpha blending. There is nothing special. Um also another nice thing about the line width is for example um the classical SVT
rendering works with integer values. Skar itself works with float values. That means we can make the width of specific lines with a float value makes which makes the scaling much more precise. For example, at this point, I hope you can see it. This is now Skia and the scaling is 150%. And now when I switch back, you will see that you still have the one pixel line
because Windows GDI just does not support the precise scaling, which would be a benefit usually. Um, also other smaller things are actually no problem at all. For example, here the line caps. Oops, sorry. Here, now it's active. You see that you do not see that much of a difference. Um, the star with specific endings of the of the lines and so on for round and so on.
Um, at the line styles, they have to say there is still a small bug when you see here the custom line style length. Um, this is not yet absolutely exact. I hope you can forgive me about that, but it's just a small remark about it. Nonetheless, I still have to say creating a 2D graphics wrapper for skier at some points is quite difficult. Um, especially then when
it comes also to scaling, for example, I hope you can also see it here. You see that the black lines are up here. And one of the scaling issues that I still have is that here, for example, the black lines are down there. nonetheless, these still small differences still exist and to be honest, some of them can't be solved in general. But I also want to mention
at this point, usually the differences between Windows and Linux are much much greater than the differences between Windows and Skia. So usually Skia is by at all animations that I show you at all images is usually in the middle of both. So somewhere between um one good example for doing for showing that is for example the curves um here for example this is now lines which are
approximated with quad quadratic uh interpolation. Um as you can see these lines have a specific length and the calculations work slightly different in Windows GDI than on Skia. These are the differences that just can't be solved in general without rewriting the code and really emulating the specific GDI layers. Um, also another quite bigger difference is the following. Uh, where is it? These are the fonts. Um sorry
that one here. H the fonts um [snorts] the fonts actually are in the wider way to some extent a problem because SVT usually does not give the specific resource of the fonts. So I have to check with Skia what's the name of the fonts and what are some of the attributes and the properties of the fonts and then Skia has to find the say the right font
in the operating system for itself. Um of course in in in in later steps of the development it might be possible to extract the fonts directly from the native widget toolkit but this is one of the draws typical drawbacks of SVT. it triples the amount of development and compared to that it's currently not worth the effort to make this run. Um and as a consequence also of
that some of these fonts are not that precise and also sometimes slightly jumps happen. I hope you can see it for here for example in the SVT layer that these jump slightly up and down. It is just not pixel exact. And I also have to say at some fonts also the the font is different one because um SVT has a legacy support for older fonts and in
some applications these will be used and ski has no chance to finding them. Even if it would be possible for some of them ski wouldn't load them because it does no longer support these formats. Um yeah so much to that. So these are this actually the state of the current development but nonetheless when you check out the features here all the features of the classical GC they
are all supported and yes at the most of them you will only see slightly differences. um also transformations for example that also means for example it would be possible to use such transformations or clipping modes and so on also then later I have to say it's not yet implemented but as you've seen previously with this with this gear bridge to the browser it would be possible then
to use these graphic engine also in the browser um yeah for example clipping also works here this is the animation of that as you can see there is in the line thickness is slightly difference but that's all of it um the this is was the comparison of that what we already have so let's check some code snippets so what's till now I've told you the maintenance would
be better the implementation is better. You have slightly differences between the the classical one. What is the real benefit of the current Skia implementation in SVT? And for that, oh sorry, I want to give you a performance This is actually the Windows GDI implementation. Um, it is actually just a snippet as you can see which draws some text and in the background I've put much much more
texts which just move from left to right and as you can see the frame rate is rather disastrous and compared to that with the this is now actually the final intention of that what I wanted to achieve. I have to say in order to achieve this rise in the frame rate from 20 to 110. Um for that I had to use tons of caches and I also
had to use all possibilities to increase the frame rate to the limit. But as you can see it is possible and one of the benefits is actually by doing that it is possible on all operating systems. That means you can maximize the performance only once and it works on all operating systems that you need and yeah this is actually then the benefit of it. So so much
to the increase of performance. Um unfortunately I also have to tell you at this point Oh, I'm not sure. Unfortunately, I also have to tell you at this point um the performance in general is faster as you've seen with the text. It is much much much faster. Nonetheless, also it's faster than Windows in general. Nonetheless, there are still some cases where Cairo and Linux are more or
less on par. Actually, the Cairo implementation is much much more performant than the Windows implementation of SVT. And in generally, often it's on par and getting better than that is quite cumbersome sometimes. But nonetheless, I have to say the potential to be much better is there, but it's just not yet fully implemented. So, let's talk about the architecture. um since this is the a direct integration of
ski into the existing SVT I was very careful what I'm doing for example when you use rap or when you use another SVT replacement bit more fe with more features. Usually, if it does not work fine for them, it's not a big deal for them. You can still switch back. And at this point, I want to prevent that um doesn't work anymore or that any kind of
bugs are created by that. So, I've made the changes to the classical SVT as less as possible. And I actually did that by just creating a fragment um with all the Skia resources in there and to the side of this this in the classical SVT layer I only created a very very small extension mechanism to use then skier in the S in the to use then gear
in the skier fragment that means actually in the canvas I've created this attribute external canvas handler and if you then use the ski style this external canvas handler will then be load loaded with the service loader if the specific fragment is then there. And the next modification which was necessary is when the specific drawing event flow comes up to the skier layer and to the SVT layer.
Then if the external canvas handler is set, I just delegated further to the GC layer and the rest of the paint handling then happens in the skier handler. That means I handled there the resources for the ski surface and prepared the actual drawing as we know in SVT we have painting events usually with a classical GC and there comes the second modification that I had to do
instead of using the classical GC for a drawing now I've created a GC extension as you can see here and this GC extension extends the GC um you probably know the GC C is actually final. I've modified it and made it sealed extend with the GC extension. And by that all the call events to draw something now go against the GC extension. And the GC extension itself
is just a delegation to the skier GC then which then draws everything that happens there directly on the skier surface. And at the end of this drawing then it just happens that you've got the skier surface with all the drawings in there and then you switch just the skier surface. So you swap the buffers in just like in OpenGL how it's done in OpenGL and then you've
got the result of the drawing on the canvas. That means at this point OpenGL is necessary and yeah this is actually then the current implementation and the current state of the implementation. yeah okay as I've mentioned earlier we are still at a very early state we still have many limitations and missing features um as I've said already the operating system and the architecture are quite um restrictive
um also that it only works on the canvas since in general in SVT you can also work um draw something on the composite um with the text layout out actually there is a Skia text layout implementation as a proof of concept it's not yet sufficient to use it directly in this ski canvas um there is a fallback to the classical text layout and that means uh the
classical text layout will be used it is also one of the points where the API is a problem as you can see for example here there is no possibility to add a style variable and make it clear that the that a new text layout about type can should be used now there the API has to be extended to support gear if you want to have something switchable
as I've mentioned it um also images are quite complex to use um usually the image resource lies in the native operating system you can get them out into the image data as a binary data and then you ex can execute a conversion to the gear image type. Um, of course, if you do that too often, you need quite a lot of resources. I've implemented a C cache
mechanism to prevent unnecessary recon conversions. Um, also about the image types, um, uh, the operating system can load tons of different image types and then you've got three different operating systems with various image types that exist there. having the from that a conversion directly to the one Skia RGBA format can be quite cumbersome and is not yet fully completed. Um also one of the next points is
also that you do not have a proper API support here for setting up ski variables here. You would also need an um the extension with the style. Um, it is would be quite a benefit to have an SKIA image type because many um applications that draw something use images as a support layer for then putting these images on the canvas. And when you then use the classical
uh GC on this image, you draw with the classical GC implementation. And this classical GC implementation must then be converged and then put to the skier canvas. And there then you have the problem that the performance is not that great and also that you have a mix up between the classical GC and the skier GC. So yeah there some more work is has to be done and
actually the rest is more or less the consequence of that um except maybe the carrot but the carrot is no not that big deal but the style text usually is just the main you probably one of the most prominent user of the text layout and by that it is also not yet supported for the classical for the ski canvas also decorations and shell um there I'm just
more restrictive and about the font support that's what I've mentioned earlier. Okay. Um so when you're interested in the project um the pull request to the SCT master branch is open. I hope we can merge it soon into it. There are still some details but then it should be done. Um you will see it then in the release notes. So if you would like to test the
gear canvas without any further modifications to your code, that means if you do not even want to modify one line of code, there are two um VM arguments that you can use. These are these two here. So when I go now here to my oh sorry tree example this is actually just a snippet from draw to D which actually works quite fine and as you can see
down here I've added these two arguments actually the first one is the force enabling of all canvases with the skier canvas the second is just the lock to make sure that actually the steer canvas will be used. It's not that easy to see the difference in many cases and as you can see here there's not much of a difference in the drawings and so on. So these
examples work quite well that means also if you have some uh GF application or some other graphics that you might want to use in generally I have to say there shouldn't be any problems. So yeah. Okay. And as you can see also the lock works with the specific information that it has been started. Um [snorts] okay then a last remark to the um to the browser support.
At this point I have to say nothing of that has been implemented now but it would be quite easy to use if with this SVT skier layer. Um there is actually a feature which is called picture recorder and ski jar which is of course just a delegate of the skier feature of it. And this one provides as you can see here a canvas and this is now
a skier canvas not the classical SVT canvas. And here at execute drawings of course you could use the skier canvas and put it into the skier GC as I've showed earlier in the in the in the architecture di diagram and there then you could just execute all the drawings now on this canvas and no longer at any other place. And the advantage of that is when you
are finished with all your drawings, you can finish your drawings and then you can serialize the these are actually the drawing instructions and you can serialize these to your binary data and then you can just use these binary data move them to your browser and with canvas cat you can just execute all these drawings once directly and the performance of that should be similar like it is
should be to the classical desktop schem integration depend dependent on that how the browser and canvas kit actually are using um um skier or how the uh web assembly u compiling has worked. Nonetheless, of course, this is just a concept idea. I just want to mention that this would be possible. Now, of course, there are still details that must be cleared. Of course, you would probably like
to have the event handling with the mouse and the keyboards event from the browser. they must be sent to SVT and so on. This all would then still be necessary to do. Okay. Um, and this is currently the state of the project. As I've said earlier, the pull request is open. I hope we can merge it soon into SVT. Um, So I also would like to thank
um some of the supporters of this project and also Haiko. And that's it from my side. So are there any questions? [applause] >> I It's a really interesting um presentation. Thanks. Um you mentioned right at the end you you hope that you this will be merged to the the sources soon. >> Yes. >> Are are you suggesting that it will be possible for people to simply switch
their existing RCP apps to use this you know sometime in the in the near term. >> Yes. Yeah. Yeah. Of course. Of course. And to what extent would you expect that to be functional then at at this stage without you mentioned keyboard and pointing device support and things? >> Yeah. At the at the at the actual point about the browser support [laughter] you you mean uh when
this will be finished with the browser support or because because of the browser support there has is this current state does not support it. This is just what I wanted to say that in the future it can be implemented and it would in my opinion be quite easy. But at the current state for me um because of the less resources that we have, I would actually currently
first of all get the ski canvas into the classical desktop widget toolkit to make this properly working for all types. And actually I would at the moment my focus is much more on getting the test coverage probably done and get the quality increase the quality also of the code and fixing bugs and then the next step would for me actually first the text layout and not the
browser >> Great. Thank you. Um yeah since you me me me me me me me me me me me me me me me me me me me me me me me me me me me mentioned G one question. So G itself has its own weaper for this uh it has it own weaper which then communicates with the SVT GC >> do you think it would make sense
to define a separate uh weaper which then communicates directly with the uh skier surface or wouldn't this make a difference in term of performance? >> Uh I don't think that would cause any performance decrease. Um it's this actually should then be discussed uh pro for for this first version actually I only want to provide something as a duplicate of the classical GC but of course I also
to be honest the complete project that I've shown till now is just the intention to make a step in the right direction with all of that and to make to to widen the scope to much more scenarios which are possible from my side I'm fully open for that >> and it was just a general idea whether you have considered this or >> uh no not yet. Uh
first of all, thanks a lot for the effort that you put into this project. Um I think it's a very important piece of work to also figure the limitations of the SWT APIs themselves. Um and your performance u scenario with the 1,00 frames per second. Yeah. and the command line argument that you showed later they they caught my attention. What would be the frame rate if you
would just have used those command line arguments instead of changing the boolean flag to true >> exactly the same. >> So all the optimizations that you mentioned to get to this frame rate they are not specific to the application but they are in the implementation of the extension. Um actually um I've implemented some VM arguments in order to remove caches for for testing reasons. That's that's all.
But um the reason for these VM arguments are just when you're just just a developer and for example you're using the G layer and you can't access the canvas that will be constructed directly but nonetheless you want to know how does my application on top of GF work with that then you can just activate these VM arguments and then every canvas constructor will expect think that you
are using this these styles and that's the only difference with these. Okay, thanks. >> Anyone else >> uh by using the VM arguments, would you be able to also hook into those hidden canvases inside of the style text for example? >> Yes. In general, yes. Um the style texts are actually I would say the master class of of of graphics implementation because they are really really difficult
and yeah but yes in general it would be possible nonetheless. Um I have to say I these are in generally disabled because it would just not work at the moment in the current state. Yeah. Nonetheless, as I've said, we there are prototypes with a ski text layout. Uh they are just not in the state to be merged. So have you rolled this out into any existing products?
Like has SAP use this internally in production yet or is this um being merged to be in production? >> No, at SAP it's not yet in production though. >> Could be done maybe. [laughter] But actually one one remark about it. I've tested it with with with SVT code as I've shown you the VM arguments. So it basically it would be possible for everyone at SVT who uses
the development tools based on Eclipse to activate these V VM arguments. Okay. So um one remark on this um because there were some concerns raised during the project already that maybe native support was dropped at some point or Patrick also asked whether directly use Skia for for example GF. Um of course the idea to integrate that now into SWT is fully optional for now right so you
can always switch back and that's also the intent of course to keep everything working as is. Uh one question uh on that I mean the initiative 31 originally aimed at replacing all widgets with custom implementations based on Skia. Now this is really focused on performant canvas rendering but there are actually some custom widgets already implemented right like capab folder for example have you tried already whether they
could be they could easily adopt ski because they just use the canvas underneath right and maybe we could uh at least adopt uh Skia for those custom widgets as well. actually because of this project I've worked quite a lot to get this project running and I have not yet tested it but I can assure you it will work because the table implementation that we've written for the
initiative 31 work with a much much worse GC ski implementation of previous so if it worked with that then it will definitely work with this >> some further potential to unlock better performance or >> yes yes of course I've also talked about uh integrating making a better table implementation. But there once again it's the question about the architecture because usually you have the table viewer and there
are many applications that use the table viewer and if you would like to create the table implementation on top of that this would mean that you have with the hierarchy. Currently the problem with the sea table is that it does in a completely other hierarchy than the classical table and when you if you want to use the table viewer with the sea table it just does not
work because the hier you you the hierarchy crashes because it's just not part of the table hierarchy but if you would create something like that it would be possible then to have a high performance table implementation in inside of SVT. Yes. But this is also just an what if. Yeah. >> Hi. Uh, thanks for your presentation and efforts. Maybe I'm going to embarrassing myself here now, but
I've never heard of Skia. So, can you tell me so or explain something about the selection process for that project? [laughter] okay. Uh, the reason why Skia actually this question can to can to some extent be splitted because why Skia? Why not impeller? It would be also completely legit if you want to have m performance than go directly to Ampella. Impella actually impella is a graphics library
as a next generation graphics library with even more power per performance like ski. Of course you could ask this questions. Um ski we I think in initiative 31 we've chosen ski because its reliability and its availability on all platforms. It's just the the big player in inside there. But nonetheless, I think it would be also be very very good possible to use this feature, this paint event
externalization of SVT to use it also for other graphics libraries. It's also to some extent always a question do you change the architecture and then try to get more features in there or do you try to create features for for for the existing architecture and then have actually a real reason to modify the architecture. So maybe we can get more, how should I say, motivation or impulses
into that and to have more external features into SVT. But this is just not my my concerns in general. You have to >> thank you.
